Yeah, my new Filco Brown has a similar slight roughness, up-to and through the tactile bump. Below that it feels more smooth, but maybe just because there is so little resistance.
It doesn't really bother me, and since it feels like rubbing of some sort, I feel like time will probably smooth it out like ripster suggests.