Okay, I play games, hit the F-keys quite a bit... also when typing numbers quickly... my hand (as well as wrist) moves slightly forward to hit the key. Also,
The problem usually occurs when I adjust back to my normal position... i.e. I bring my wrist back a bit. While doing all this, part of my palm is still on the wrist rest. Therefore there's some lateral force/friction.
Not all times, but sometimes.... I just take the wrist rest back with me, causing it to slide.
My desk is wood, painted with a clear-coat on it if that is at all relevant.