It is good to observe common ESD-safe practices as you say you already do. That should be enough. I have built computers wearing non-plastic clothes with a concrete floor and zero ESD protection and gotten away with it. One of the guys in one of my previous (electronics repair) shops where I worked would jokingly say "ESD is a myth!" when someone started making a fuss about the ESD mats having a sheet of paper on them or some similarly overly-anal thing.
You're doing fine. You don't have to go nuts. But if you work over carpet, in plastic shoes, plastic socks, and plastic clothes, on a dry winter day . . . have a spare controller handy for when your keyboard dies!