This is an issue that is familiar to me (as well as happy merchant, Zeal)
Here's a funny story:
Zeal was sending me some springs (and a massive pile of. Kit Kat) so I asked him if he could toss in a gateron as well so I could see what it was like.
He sent me one freaking switch and it was a defective switch which had the issue that you had.
Zeal knows about this and actually ended up desoldering his board to get rid of these faulty switches. I apparently got one out of his **** pile. I guess the one zealio switch I got from him made up for that.

I believe this is a problem with the stem since I noticed the issue when transferring the gateron stem I had to a basic MX housing.
Just toss the defective switches out,or at least the stems. Most of them should be fine but some do have this issue.