USB KVM switches are supposed to not have any latency. Try it, if you don't notice anything then it should be fine.
Avoid IP based KVM switches however, also known as iKVM for some retarded reason. IP/network based KVM switches _do_ introduce latency.