IME the most significant contributor to differences in switch feeling is actually the keycaps. Remember that the keycaps are in fact, sprung weight and not free floating. The weight of the keycap can reduce the force required to actuate the switch by virtue of their weight.
Which is why if you go too heavy, the switch doesn't come back up. Perfect example of this is the Chicony super-thick double-shots, which are so heavy that they can self-actuate some MX Reds and usually do not return with MX Blues.