Nah, unnecessary drivers don't even get loaded. At worst, it takes up space on your hard drive.
The only possible issue is if the OS loads it by mistake, and it interferes with the default driver. But if you haven't noticed any weirdness yet, that isn't an issue.