ATmega32u4 partially working

[uberben]

I'm working on a custom keyboard that uses an ATmega32u4 and MCP23018 IO expander, similar to the Ergodox. My firmware is TMK based. I got to the point where all the keys connected directly to the 32u4 were working, so I wanted to add the I2C code. When I added this code, the keyboard started acting weird and caused my computer to start acting up too, perhaps I had inverted some signal lines so all buttons appeared pressed?

I removed the I2C code and reverted to what had been working previously. Now my keyboard is sending incorrect key codes over USB. If I change my keymap, the incorrect codes change, but they are still incorrect. I swapped out my 32u4 dev board for a spare and flashed the same old, working code and sure enough things are working again.

Does anyone have any ideas how a 32u4 could start acting up like that? Is there any way to recover from such a failure?

[DrHubblePhD]

Download EK switch hitter and see which inputs are actually being detected. Obviously check your soldering to make sure the chip is not backwards or improperly soldered.

[uberben]

It looks like EK switch hitter is Windows only. I was using the keyboard viewer built into Gnome on Linux. It was showing everything from letters to numbers to symbols to media keys, even a couple that looked like macros (multiple keys pressed simultaneously).

Soldering is not an issue. I'm using an Adafruit 32u4 breakout board plugged into a breadboard. Removing the one dev board (which I've been using without issue for month) and replacing it with a second shows that the rest of my wiring is still good and the issue is isolated to the single 32u4 breakout board.

[vvp]

Maybe you really damaged your ATmega32u4.
If you want to look what is really sent over USB wire in linux then use usbdump.

[uberben]

Maybe you really damaged your ATmega32u4.

That is what I'm afraid of. Not that I have tons of experience with micros, but I would have thought breaking the chip would cause more of it to stop working (USB, IO, etc.). It is also strange that just working with the I2C registers could kill it, unless there is a slight chance the IO expander shorted power to ground through the switch matrix (though even that I'd think would kill the IO expander and/or trip the fuse on the dev board before affecting the 32u4).

If I truly did damage the board, I'm pretty nervous trying again with a second dev board. I don't want to accidentally fry a second one. I'd like to get to the root of the problem before trying again so I can avoid that.

If you want to look what is really sent over USB wire in linux then use usbdump.

Thanks for the tip. I'll check that out later today.

[metalliqaz]

I doubt that the ATmega32U is actually damaged.  When I was developing my Epsilon code I did all kinds of stupid things and never managed to damage mine.

There are really only three permanent changes that you can make to the microcontroller.  One, program flash memory.  This is what you do when you load code and as long as the bootloader is working, you're fine.  Two, program EEPROM.  Unless your code makes use of it, then it doesn't really matter at all what's in there.  Three is the embedded fuses.  These are where you can screw things up.  In the entire length of my keyboard projects, I never had a reason to change any of the fuses.  If you changed them around while trying to get your I2C to work, then it is conceivable that something changed that won't be fixed by loading old code.  However, It's pretty unlikely that someone could change those without being aware of it.

My advice is to dive into the code.  Check everything to make sure you're running the code you expect.  Start with the most basic of programs to verify that works correctly, then build from there.  That's what I did.  I'd have issues where nothing seemed to work, and I'd have to start with an LED blinking program, then add matrix scanning, then USB, etc.  It can be a real pain in the arse to debug an embedded project without an ICE debugger.  That's why as soon as my code was working well enough, I added a debug module that let me record events and examine memory.

[uberben]

Thanks for the encouragement metalliqaz.

Like you said, fuses probably didn't change as I didn't change my flashing command or anything.

I could drop back to a blinky project and work my way back up, though the fact that my full code base works on one dev board but not the other is worrisome. I'll see if I can zero the EEPROM and maybe the non-bootloader portion of the PROGMEM. The partial operation of my board makes me optimistic that I'll eventually get it back up. Perhaps if nothing else, building up the code base from scratch again will pinpoint what part of the chip is broken (plus give me better familiarity with the code base).

[vvp]

ueberben might have e.g. start I2C while his MOSI or SCLK was shorted to GND or Vcc. Playing with micro controller IO pins can damage it if the pins are connected to GND or Vcc. If they are disconnected then the only way you can "damage" it is by disabling firmware update with fuses. IIRC you can still "un-damage" screwed up fuses with a 12V parallel programmer. But that is not typically available at home.

Anyway if changes to hardware are not done then it is often easy to check whether something was damaged. Just load the last know correctly working firmware and check whether it still works.

Edit: Oops, that is I2C, not SPI. So it can break only when SDA or SCL is connected to Vcc.

[metalliqaz]

Thanks for the encouragement metalliqaz.

Like you said, fuses probably didn't change as I didn't change my flashing command or anything.

I could drop back to a blinky project and work my way back up, though the fact that my full code base works on one dev board but not the other is worrisome. I'll see if I can zero the EEPROM and maybe the non-bootloader portion of the PROGMEM. The partial operation of my board makes me optimistic that I'll eventually get it back up. Perhaps if nothing else, building up the code base from scratch again will pinpoint what part of the chip is broken (plus give me better familiarity with the code base).

Yeah that's definitely strange that one works and the other doesn't.

Have you gone and double checked that the expander was wired correctly and that the I/O pins were initialized correctly?

[vvp]

I'll see if I can zero the EEPROM and maybe the non-bootloader portion of the PROGMEM.

It is a good idea to zero EEPROM even when doing normal firmware update. You do not need to bother with unused PROGMEM. If the firmware you are loading is working and does not use some parts of PROGMEM then it should not ever even read them (so it does not matter if they contain some old data). Although I'm not sure it is possible to load a new firmware without cleaning the whole PROGMEM first.

[uberben]

I did a little bit of investigation.

I found how to read the fuse bits from the boards: avrdude -p m32u4 -P /dev/ttyACM0 -c avr109 -U hfuse:r:high.txt:s -U lfuse:r:low.txt:s

I read the fuses from the broken board and the working board and they are exactly the same.

I ran usbdump on the broken board. If I press the key that is supposed to be 'e', I get a '.'. The same line appears in usbdump for my actual keyboard when I press the '.' key.

If I press the key that is supposed to be mapped to '5' on the broken board, I get several buttons pressed. These lines were captured by pressing and releasing that one button:

Code: [Select]

108.504832 1<-- 8: 0100 0000 0000 0000
108.512831 1<-- 8: 0100 0800 0000 0000
108.520829 1<-- 8: 0100 0800 0000 0000
108.528831 1<-- 8: 0100 0805 0000 0000
108.536828 1<-- 8: 0100 0805 7500 0000
108.544828 1<-- 8: 0100 0805 7506 0000
113.984874 1<-- 8: 0000 0805 7506 0000
113.992892 1<-- 8: 0000 0005 7506 0000
114.000874 1<-- 8: 0000 0005 7506 0000
114.008872 1<-- 8: 0000 0000 7506 0000
114.016881 1<-- 8: 0000 0000 0006 0000
114.024873 1<-- 8: 0000 0000 0000 0000

I'm not really sure what else to do with usbdump at this time.

I programmed the working code to the dev board again and it does say:

Code: [Select]

avrdude: NOTE: "flash" memory has been specified, an erase cycle will be performed

I haven't checked to see if this does both EEPROM and PROGMEM, but it looks like at least PROGMEM is being erased each time.

SDA and SCL were connected to 4k7 pull up resistors. I just confirmed that none of my wiring was shorting any of the pins together. I don't recall if I had any buttons pressed though when the problems occurred. I need to read the data sheet to see if I set the pins in the wrong modes or to the wrong values, but so far I'm not seeing any way I could have created a short unless I had set a column to drive high and a row to drive low AND also pressed a button.

[uberben]

It's alive!

I realised that, if I could read the fuses from the good and bad boards, why not read the EEPROM from the good board and flash it onto the bad board?

Code: [Select]

avrdude -p m32u4 -P /dev/ttyACM0 -c avr109 -U eeprom:r:working.eep:i

Then I swapped boards...

Code: [Select]

avrdude -p m32u4 -P /dev/ttyACM0 -c avr109 -U eeprom:w:working.eep

Now everything is working again as expected

Thanks everyone for helping out!

[clickclack123]

Just curious, why would the EEPROM being different make a difference?

Does TMK read layout info from the EEPROM?