It's all in your head . Rougher surfaces collect more sweat, grime, and dust--they just conceal it better by not feeling greasy as quickly.
That rougher surface is indeed collecting more grime; which means it's NOT on your fingers, because it has been "collected" by the rough surface of the keycap (which you are still touching anyway...).
The smooth keys allow that same grime to remain on your fingers, rather than sanding it off for you automatically as you type. It then collects on the edges of the cap surface, and at the edges of your fingertip-pad/contact point, and absorbs into your pores... which is why it "feels like there's crap on your fingers." There
is! (but you're still touching that icky grime, regardless of whether the surface is rough or smooth!)
If you don't sand the surface, the impurities remain on said surface. If you do sand the surface, the impurities "collect" within the grit of the roughness.
You're still touching ickiness with ickified fingers, regardless.
You should thoroughly clean every contact point in your entire environment, including (especially) your PC peripherals, and then insist on thoroughly washing/scrubbing your hands, prior to using your interface devices. Stuff will
still get gunked up, gradually... but you can significantly mitigate and minimize this, which, A) keeps your stuff un-yucky longer, and B) extends/prolongs your enjoyment of your various contact surfaces, while, C) also minimizing the potential vectors for pathogens to be spread among contact-point-sharing individuals.
This is why i despise people who refuse to properly wash their hands after taking a dump, or having a crotch-scratch-fest: they then proceed to invisibly contaminate EVERY shared contact point (doorknobs, fridge handles, game controllers, TV remotes, etc.), which then requires me to be inevitably exposed to whatever germs they've acquired elsewhere, through such careless and unhygienic practices. I can't run around cleaning up after everyone all the time. I have things to do; even when i don't have things to do!
But yeah. I'll just mention Deck's nearly indestructible backlit caps. People seem to hate the font/legends, but i don't see it as a bad thing; just different. Nothing inherently wrong with "different" (unless in the context of social conditioning, and the fact that herd-think tends to cause people to fear and reject anything perceived as "different;" which i don't THINK should apply to keycaps... but probably kinda does, because that's how people are).