Help!! Ergodox Thumb Keys Aren't Working!

[clickclack123]

Well, yesterday i got my Ergodox, thanks czarek! It certainly looks the business but…

Unfortunately, the thumb keys on the left hand side aren't working.    When I first plugged it in and loaded the ergodox-kinesis-mod hex, it kept going to the screen you get when you hit ctrl-alt-del over and over.

If I start it with the left hand disconnected, the right hand works fine by itself.

Actually one time I jiggled the cable a bit and both hands worked for a while but at that time none of the left thumb keys did anything.

Then I made my own layout using the massdrop ergodox configurator with just letters on the thumbkeys, this stopped it from going to the ctrl-alt-del screen, but promptly hard-reboots my pc every time!! I think Win7 completely barfs when zillions of keypresses are sent to it. Who knew! Might be a good trick to play on coworkers or people you hate (or even better, coworkers you hate).

Basically I think that there must be a solder bridge somewhere that is making it think that all the thumb keys on the left side are being pressed at once. In the 10 seconds before it crashes the machine, I can see that the left thumb characters were being held when I ran the Aqua key test program.

I don't have a full schematic of the Ergodox, the only one that I could find was a very simplified one without the actual key connections on it, just the teensy and TRRS/IO expander.

Anyone know where I should start checking the board for faults? I assume on the left board, near the thumbkeys… I'm thinking that there must be one place where, if bridged, it will confuse the IO expander into thinking that all the thumb keys are being held at once.

Does anyone have a full schematic of the ergodox? I haven't attempted to install software to open the pcb files yet, not sure if that will help me a huge amount with this problem anyway.

[plainbriny]

Well, yesterday i got my Ergodox, thanks czarek! It certainly looks the business but…

Unfortunately, the thumb keys on the left hand side aren't working.    When I first plugged it in and loaded the ergodox-kinesis-mod hex, it kept going to the screen you get when you hit ctrl-alt-del over and over.

I had a similar problem before, but not exactly identical. My problem is that a solder joint is broken, which make my left thumb cluster non-functional. I have to use a wire to re-establish the connection.
In your case, as you said, it is likely some sort of shorts. If possible, you can get a multimeter to check the resistance between the solder joints.
Also, maybe you can post a photo of the backside of your left thumb cluster? It might help in some ways.

[clickclack123]

Well, yesterday i got my Ergodox, thanks czarek! It certainly looks the business but…

Unfortunately, the thumb keys on the left hand side aren't working.    When I first plugged it in and loaded the ergodox-kinesis-mod hex, it kept going to the screen you get when you hit ctrl-alt-del over and over.

I had a similar problem before, but not exactly identical. My problem is that a solder joint is broken, which make my left thumb cluster non-functional. I have to use a wire to re-establish the connection.
In your case, as you said, it is likely some sort of shorts. If possible, you can get a multimeter to check the resistance between the solder joints.
Also, maybe you can post a photo of the backside of your left thumb cluster? It might help in some ways.

Thanks for the reply. I'll post a photo tonight when I get home.

What I'd really like is a good quality picture of the Ergodox circuit board, like this one below, so I can see where the tracks on both sides connect to.

I want to check it with a multimeter, but it's pretty hard to see where the tracks go on the black soldermasked pcb that I got from czarek (even though IMO it looks nicer than the blue pcb).



I haven't had any experience with pcb software, and also now I have to go through the slow phone for internet until Friday thanks to stupid bloody workmen and their clumsy backhoes!

Does anyone know where I can get a good quality pic of the pcb, or could a pcb guru export me a good quality pic of the pcb with the tracks on both sides visible please?

[plainbriny]

check the attached pdf file, hope it helps

[clickclack123]

check the attached pdf file, hope it helps

Awesome! Thanks so much, I'll check it now.

I put a few photos of the thumb part of my board up at https://plus.google.com/photos/113640154161030352630/albums/5976437170917991057.







I couldn't spot anything but I'll have a prod around with a multimeter now.

[plainbriny]

check the attached pdf file, hope it helps

Awesome! Thanks so much, I'll check it now.

I put a few photos of the thumb part of my board up at https://plus.google.com/photos/113640154161030352630/albums/5976437170917991057.

Show Image

Show Image

Show Image

I couldn't spot anything but I'll have a prod around with a multimeter now.

SW0:7 looks suspicious to me.
BTW, if your switches already have diodes, you don't need to solder extra 1N4148 diodes.

[clickclack123]

SW0:7 looks suspicious to me.
BTW, if your switches already have diodes, you don't need to solder extra 1N4148 diodes.

Working!!!

Took all night, but I ended up desoldering every left thumb switch except one, starting at SW0:7. That wasn't the one that was the problem, I think one of the others may have had a little too much solder on it and shorted to the ground plane on the top. Installed them individually back again, testing that they weren't shorted again at every step, and it worked in the end!!

Now to get used to this crazy thing!!

[clickclack123]

Damn, seems I spoke too soon.

Last night it worked for about 3 hours, then started doing the same thing. I ended up desoldering and removing every left thumb switch again, then installed them all back. Then it worked for about 30 minutes, then started doing it again.

This is driving me crazy!! Plus I'm worried about damaging the tracks on the board from all the desoldering/resoldering. Now I'm worried that the IO Expander is flaky, although it obviously does work fine on all but one pin (the one that connects to the thumb switches).

I measured the voltage on that pin on the IO Expander, and it is about 2.2V, when all the other rows are at 5V. So it's being held lower than it should for some reason. I guess there's a short but damned if I can find it.

Stay tuned!

[clickclack123]

...and, after hooking up the multimeter and looking around, now it's working. 

I didn't change anything, just poked around.

The red part is at 5V now, it was around 2V yesterday when it wasn't working.

[clickclack123]

So, the Ergodox worked perfectly all night last night. For about 5 hours.

...Then I get home from work today, and it's not working. The pc is seeing all the left thumbkeys as being held down. I measured the red part from the diagram, and it's at 1.5-2V again!

Anyway, I'm now blaming the IO Expander. I just shorted out pin 8 of the IO Expander (the red part) with the 5V line from the TRRS cable for a second, and now the keyboard is working fine. It seems that for some reason that output line can't get to 5V without a little help!

My girlfriend just said that she can't believe that I've built a crazy keyboard that I have to jump start! 

But I'm just stoked that the thing works and I don't have to spend the night soldering again.

Anyone know where I can get a replacement mcp23018 in Australia so it doesn't take too long to get here? Digi-Key Australia wants $34 for postage on a $3 part! RS Components has reasonable shipping but has a minimum order of 5.

[clickclack123]

Well, today it did it again, and I had to jumpstart it. At least I know what's going on now. It works perfectly after the jumpstart, even if you disconnect the usb. So weird...

Loving the Ergodox. There's definitely a learning curve though.

[AKmalamute]

Anyone know where I can get a replacement mcp23018 in Australia so it doesn't take too long to get here?

Newark carries the device, and since they ship some parts out of England it might not be quite so bad as other places, but their shipping tends toward the high end too -- but still less than some other online shops.

I'd offer a link to their catalog but their site isn't responding right now. ...good luck...?

[clickclack123]

I just ordered 5x mcp23018 from rs components for AU$15.

That was even less than Newark (Element14) wanted for a single one with $13 shipping. 

And don't even mention Digi-Key!! $34 to send a single chip!! 

Jumpstarting the keyboard now no longer seems to work for longer than a minute. So I think I'll be without an Ergodox now for a few days at least. It's been a bit of a saga.

[MOZ]

Are you sure it is the I/O expander

[clickclack123]

Are you sure it is the I/O expander

I really don't think it could be anything else, because:

1. The other keys work, so the trrs must be working because the IO expander is correctly sending keys to the right hand side, through the trrs. If I make a layout where the left thumbkeys don't do anything, the rest of the keyboard works perfectly all the time.

2. Up until recently, I could just jump start that pin on the IO expander, and the board would work for hours after that. This makes me think that it can't be a short or anything wrong with the switches/diodes because how would jumpstarting it fix that?

3. The only point where all the thumbkeys meet is at pin 8 of the IO expander, which is at 2V when they are held, and 5V when everything is working.

So in my mind that only leaves the IO expander.

But I'm open to other ideas.

[justnits]

what about the diodes' solder point?
seems like they were touching together...

[clickclack123]

what about the diodes' solder point?
seems like they were touching together...

Sorry, I'm not sure which point you mean?

[justnits]

what about the diodes' solder point?
seems like they were touching together...

Sorry, I'm not sure which point you mean?

my bad, i zoomed in further into the pics and saw the solder point didn't "flow" across to the other side.
what about the contact point on the teensy?

[clickclack123]

what about the diodes' solder point?
seems like they were touching together...

Sorry, I'm not sure which point you mean?

my bad, i zoomed in further into the pics and saw the solder point didn't "flow" across to the other side.
what about the contact point on the teensy?

I think there's no way it can be the teensy, or anything on the right hand board. The right hand always works perfectly. It's only the left-hand thumb keys that are flaky.

[clickclack123]

OK, I replaced the IO Expander yesterday... and...

... it still doesn't work!! 

Exactly the same, it acts as if all six left thumbkeys are held down, until I jumpstart pin 8 of the IO Expander to 5V, then it works for a while, then back to broken again!!

I replaced all the diodes in the left thumbkeys with new ones as well. I'm pretty stumped tbh, and I have been obsessed with this stupid thing for weeks now...

Can someone running hasu's firmware test something for me? When you run hid_listen, and hold down LShift+RShift+X, it gives you a scan of the matrix of what keys are held down. When my thumbkeys are NOT working properly, the output is like this:

Code: [Select]

r/c 0123456789ABCDEF
00: 0000010100000000
01: 0000010100000000
02: 0000010100000000
03: 0000010100000000
04: 0000010100000000
05: 0000010100000000
06: 0000010100000000
07: 0000000000000000
08: 0000000000000000
09: 0000000000000000
0A: 0000000000000000
0B: 0000000000000000
0C: 0000000000000000
0D: 0000000000000000

When they're working properly, it's like this:

Code: [Select]

r/c 0123456789ABCDEF
00: 0000000100000000
01: 0000000100000000
02: 0000000100000000
03: 0000000100000000
04: 0000000100000000
05: 0000000100000000
06: 0000000100000000
07: 0000000000000000
08: 0000000000000000
09: 0000000000000000
0A: 0000000000000000
0B: 0000000000000000
0C: 0000000000000000
0D: 0000000000000000

It seems strange to me that there is a column of 1's even when the thumbkeys are working (Column 7). What do you see on this output when there are no keys pressed?

[MOZ]

Did you install the filter capacitor?

[clickclack123]

Did you install the filter capacitor?

I did have it on before, it is now removed though.

[MOZ]

I would put it back.

How much is the voltage on thumb row pin now?

[clickclack123]

I would put it back.

How much is the voltage on thumb row pin now?

I'll put the capacitor back on tomorrow. It's 2:30am here now, gotta get some sleep soon. 

The voltage is really low, about 40 to 120mV atm, and the thumbs aren't working.

...Just shorted it to the 5V, and now it stays at 5.12V, even with the jumper removed.

It's so weird, if I short that pin with 5V, it will work perfectly for a while, all the switches will work. Maybe for 10 seconds, maybe for hours, but they will work. That's what makes me think it can't be a short somewhere on the board. If it was a short, then why would using a jumper switch it back into working mode? Presumably the short would remain there, preventing it from working??

[wuqe]

Here's what mine looks like "at rest".

Interesting, looks like mine has the same "live line" on the right, but one on the left, too. Could it be that your columns 5 and 6 are swapped or crossed somehow? I'd focus on that area, but then maybe that's where you've already been looking...

[tlem]

Mine looks like wuqe's.  It spams the window with

[clickclack123]

Thanks a lot guys. Still working on it. I feel like Sisyphus.

[utku]

good luck on the fix, i'm watching this thread for possible fixes.

i have it worse on this batch of ergodox (it's my second one). my left hand is not working at all. and what's worse is my right hand also is faulty, i plug in my working ergodox's left to it and nope, it doesn't work either. i'm guessing that i've killed the pcb's while soldering those damn smds. i was content with old style big ass resistors, why they didn't include them escapes me.

[clickclack123]

good luck on the fix, i'm watching this thread for possible fixes.

i have it worse on this batch of ergodox (it's my second one). my left hand is not working at all. and what's worse is my right hand also is faulty, i plug in my working ergodox's left to it and nope, it doesn't work either. i'm guessing that i've killed the pcb's while soldering those damn smds. i was content with old style big ass resistors, why they didn't include them escapes me.

utku, please start a new thread describing your issue and I'll try and help you there. I'd just rather we didn't confuse my issue with your one.

Could be a problem with the trrs socket/cable on the right side?

You need to try a hex file where the ergodox doesn't care if the left side is connected or not, get the right hand side working first, then deal with the left. I can make one later but I'm snowed under dealing with my own issues atm.

One thing that I've found useful (but still hasn't solved my problem) is making a layout using tmk fw where LShift, RShift, and X are all on the right side, so I can access LSFT+RSFT+X to get the matrix debug screen up, even when there are problems with the left side. I can give you a hex with that if you like when I get home, at work atm.

[clickclack123]

...back to my problem.

I just can't work this out. It just won't work indefinitely. When I jumpstart it, it will work fine for a random time, over 4 hours last night... When it stops working, however, it will almost never correct itself without me jumpstarting pin 8 of the IO expander manually.

I've replaced the IO Expander with a new one, removed and resoldered all the thumbswitches on the left hand, replaced all the diodes on the left hand, tested and put  the capacitor back on and spent two weeks now poking around with the multimeter trying to find a short. It's just crazy.

I've tied the multimeter to pin 8 of the IO expander and checked using the continuity buzzer to every other pin on the board, and couldn't find any shorts.

Here's what mine looks like "at rest".

Interesting, looks like mine has the same "live line" on the right, but one on the left, too. Could it be that your columns 5 and 6 are swapped or crossed somehow? I'd focus on that area, but then maybe that's where you've already been looking...

My rows in the debug look just like yours now (when the thumbs are working), by the way.

Code: [Select]

r/c 0123456789ABCDEF
00: 0000001100000000
01: 0000001100000000
02: 0000001100000000
03: 0000001100000000
04: 0000001100000000
05: 0000001100000000
06: 0000001100000000
07: 0000000000000000
08: 0000000000000000
09: 0000000000000000
0A: 0000000000000000
0B: 0000000000000000
0C: 0000000000000000
0D: 0000000000000000

I think columns 6 and 7 are on "1" because aren't they the ones where the left led's are connected to when you do the left led mod? cub_uanic mentioned in his thread that his port of the TMK firmware for Ergodox supports the left leds.

They are probably in a weird floating state because they are meant to be outputs for the led's, not inputs for switches. Sometimes the whole of column 6 or 7 or both change to "0" if I touch the pins of the IO expander with my fingers while it's running, but they pretty much always are the same as yours when you first plug the 'dox in. I must have been touching the pins and made column 6 go to "0" before in the output that I posted.

[plainbriny]

Maybe you can post a picture of the front side of the left thumb cluster?
Without the caps of course.
I suspect there might be some problem with the diodes.

[clickclack123]

Maybe you can post a picture of the front side of the left thumb cluster?
Without the caps of course.
I suspect there might be some problem with the diodes.

I've replaced the diodes and installed the new ones on the underside. I don't think it'll help too much looking at the top but I'll try to take some photos tonight. The underside is looking pretty ugly now with all the rearranging - I've desoldered the switches and put them back on three times! I'm worried any more will risk lifting the pads from the board.

[MOZ]

Could you try without the left LED mode and see what results you get?

[clickclack123]

Could you try without the left LED mode and see what results you get?

How do I turn that off?

I'm not too worried about that really as two others have said that they also have the 1's in those columns. Those two columns don't seem to have any relation to, or effect on column 5 which is those pesky thumbswitches.

[clickclack123]

So the Ergodox has been working all night tonight, which has been nice.

I've actually come up with something that will help me, even without having the thing completely working. I have it set so one of the thumbkeys switches to a layer with the whole left hand off when pressed. That will hopefully make it not crash the Win7 PC I'm usually using when it does go haywire, plus I have it set to turn the teensy led on when that happens, so it will be obvious straight away... We'll see anyway.

[utku]

@clickclack123 so was it the resoldering of diodes from the scratch that fixed it?

[clickclack123]

@clickclack123 so was it the resoldering of diodes from the scratch that fixed it?

I don't think it's actually fixed. A few times it's worked all night. Usually the next day it's faulty again until I jumpstart it again. So weird…

I'll see how it goes when I get home tonight.

[plainbriny]

I am still curious, that from the pictures you posted, you seemed to solder two diodes for each switch...
I think a picture of the front side would clarify that, maybe I misunderstand something?

Also, if your switches come with diode installed, the polarity would be critical. However, the rest of the left side (other than the thumb cluster) is functioning normal, so this might not be the case.

[clickclack123]

I am still curious, that from the pictures you posted, you seemed to solder two diodes for each switch...
I think a picture of the front side would clarify that, maybe I misunderstand something?

Also, if your switches come with diode installed, the polarity would be critical. However, the rest of the left side (other than the thumb cluster) is functioning normal, so this might not be the case.

Very observant! That was from a mistake that I made before, that I mentioned in another thread:

Thanks again wuqe. I think I've done something stupid! 

The switches that I'm using were harvested from a cheap Cherry MX Board 2.0. I bought this board so I'd get the low-profile caps basically for free for the cost of the switches. The place where the diodes are meant to be just have staples in them. They didn't even spring for the cost of diodes! I kept them in there just for physical support of the switches, not really thinking about the fact that this would effectively short out the diodes! I'll give all the staples a snip when I get home tonight. Might lube the large thumbswitches while I'm at it.

Actually I wonder if this might also be causing some other problems that I'm having.

I have opened all the switches and snipped the staple, so they are not shorted now and nkro is working perfectly.

Thanks for looking though.

BTW I also got 4 cherry stabilizers on that board as well, and the thumbswitches feel much smoother and nicer with them on... When the thumbswitches are actually working, that is.

[clickclack123]

Last night when the thumb switches were not working, I actually unplugged the left hand and took measurements of the resistance/diode voltage drop between pin 8 of the IO Expander (the thumb column) and all the other pins. I had a picture of it up in Photoshop and was preparing to post it here when the machine crashed and it was lost. 

Anyway I am suspicious that it may be something related to the pullup resistor inside the IO Expander? I'm pretty hazy on how pullup resistors work, and whether or not they're meant to be activated when the pin is set as an input. I'll do some investigation about this later on.

Maybe the IO expander is getting confused and activating it when it shouldn't? When the thumb switches are not working, I'm pretty sure the thumb switch is measurably different to the other pins. When the thumb switches are working, the resistance/drop between pin 8 and the other columns is unmeasurable.

Next time the thumbs stop working (probably tonight I'd guess), I'll take some measurements again.

[clickclack123]

I took some photos of the board. The large fullsize originals are up at https://plus.google.com/photos/113640154161030352630/albums/5982035848328321809.

Please don't judge my soldering by this board.  This board and I have been to hell and back. 

Those pesky thumbswitches:




My professional removal of the first IO Expander. 


The board looks a bit worse for wear but I have tested it like crazy for continuity and shorts.


More photos at https://plus.google.com/photos/113640154161030352630/albums/5982035848328321809

[clickclack123]

OK, tonight after reading about how pull up resistors work, I decided to add one to my thumb column so it will pull pin 8 of the IO Expander towards 5V, to simulate what I have been doing to jumpstart it to 5V to fix the thumbs every time. So far it's been a couple of hours and the thumbs have been working the whole time, so fingers are firmly crossed over here.



I added a 10kΩ between pin 8 of the IO Expander and the 5V pin from the unused USB port on the left hand.

I have no idea why my Ergodox needs this resistor, the mcp23018 is meant to have a built-in pullup resistor. Remember this is the second IO Expander that I've put in there, both had the same problem. Maybe my tmk firmware is disabling the pullup on that column for some crazy reason?? Actually it was doing the same thing with the configurator fw as well.

The best thing I've done in terms of dealing with the thumbs not working was to set up a layer that has the left thumbs deactivated that is actually activated by one of the thumb keys. I made it light the led on the Teensy when it switches to that layer, so I know immediately if the thumbs have a seizure, when the seizure is over, and an added bonus is that it no longer crashes my Win7 machine because the thumbs are no longer sending lots of keystrokes to the pc.

[MOZ]

This is interesting, did you try load ben's firmware instead of TMK?

[clickclack123]

This is interesting, did you try load ben's firmware instead of TMK?

The Massdrop configurator was the first thing I used, I believe this is Ben's fw?

My board had this problem with either fw.

BTW the Ergodox worked perfectly all night last night, for about 5 hours. I'm trying not to get my hopes up too much, but the magic resistor may have fixed it!

[MOZ]

This is very interesting, as I believe this is the first case where we are seeing this problem.

[clickclack123]

Yeah, I have no idea why my board is so weird. I'm 99.99% sure there's no short with all the testing I've done, and even without the resistor it would work for 5h at a time, then stop for no apparent reason.

Still working perfectly since I added the resistor. About 9 hours use. Fingers still crossed.

[clickclack123]

Just an update, the Ergodox has been working very well since the resistor was added. The thumbs did freak out once but after a jumpstart they have worked perfectly, so one jumpstart in about 25 hours use isn't too bad, much better than before the resistor.

Still slightly annoying that my board is weird and I've had to allocate one key to thumbswitch disabling though. I've had to leave the top part of the case off always because if the thumbs do freak, I'd have to open all those screws.

[plainbriny]

Just an update, the Ergodox has been working very well since the resistor was added. The thumbs did freak out once but after a jumpstart they have worked perfectly, so one jumpstart in about 25 hours use isn't too bad, much better than before the resistor.

Still slightly annoying that my board is weird and I've had to allocate one key to thumbswitch disabling though. I've had to leave the top part of the case off always because if the thumbs do freak, I'd have to open all those screws.

Maybe you can add a 'jumpstart' switch? This would be handy at times.

[clickclack123]

Maybe you can add a 'jumpstart' switch? This would be handy at times.

Yes I thought of this but it's just annoying to me that I'd need it at all. 

[clickclack123]

Just an update:

My Ergodox has been working almost perfectly for the three weeks now since I added the pullup resistor. It did have a seizure twice in the first week since I added it, but hasn't done it since then.

I'm almost confident enough now to put the keycap back on my "seizure" thumbswitch, and reenable that switch into normal operation.