Thanks to the good advice, hot glue seems the way to go for an individual soldering it's PCB at home.
It is just too bad that Kailh didn't thought about this potential mecanical problem, I do love there hotswap approach otherwise.
My PCB does not suffer this problem as of now and I played quite a bit inserting and removing switches already.
I may be paranoiac, or lucky
That being said as I soldered them by hand using a soldering iron (and not my heat gun) I put quite a bit of solder on them, at least much more than doing it by reflow soldering.
I did a few tries with reflow soldering and lead free solder paste and saw that to melt the solder I had melted the socket plastic casing a little bit also.
This plastic does not seem to tolerate high temperatures (260-270°C range), maybe ABS?
Maybe PCB manufacturers have to rely on low melting point solder paste in order not to damage the socket.
These are kown to be much more brittle than standard ones, that could explain why sometimes the socket pops out.