I use both QWERTY and Colemak. I hardly have any trouble typing QWERTY anymore, except for having to look at the keys in the start... and my fingers feeling somehow sullied.
I haven't taken to ergo boards either, for the same reason as you. I did other, software-based ergonomical solutions instead, such as moving the ZXCVB keys one step to the left and using the CapsLock as a scan code remapping modifier to get arrow keys and a bunch of other good stuff to my home position.