OK -- finally have had time to put a report together. My apologies for the delays; life is kind of chaotic lately. I'll skip over some of the initial examination (tl;dr: switches are pretty clean in the short timescale) and get straight into the heart of the problem:
Here's the setup: Scope hooked up to a Teensy, probing the switch and the uC output.
Looking just at the janky setup. Not a snowball's chance I'd do this for work, but these are slow signals with a pretty good voltage swing.
With one keypress, we can easily see the problem: Early contact - 68ms before the main closure. That's WAY out of spec; there's a 5ms window for bounce; it is supposed to be stable after that point. Yellow trace: Switch. Green Trace: Microcontroller detection.
Here's a different switch, showing not only an early closure (just over 70ms prior to main switch activation), it also shows a second activation after release. Mind you, these switches are unmodified, unsoldered, straight out of the shipping container. There's not a single controller that would think this is only one keypress. (Ignore the purple trace; it toggled every time through the main loop; I was curious.)
With the stems at the lowest part of the tactile dip, you can see that the switch on the right (the good switch) is still open, while the switch on the left (the chattering switch) is CLOSED.
This is what the switch SHOULD look like at this point -- there's a small gap, so it's still an open circuit.
And here's what a chattering switch looks like:
That difference is due to two things. First, the length difference between the front of the stationary contact mount and the rear of the slider guide. This accounts for up to 0.002 inches in my switches - smaller is better here:
Secondly, the slider itself can vary slightly -- there appears to be a range of 0.003 inches in my switches. Bigger is better here:
... So that's the problem. That's why hamfisted soldering can cause this. That's why you can see this on factory-fresh switches. That's why strict's fix works. And that's why you'll never see this on linear Zealios.
(I also put this on imgur so I'd have some time to write things up and try to present it all logically. Some extra photos and setup and fluff at the link:
http://imgur.com/a/dLD6I )
