I'm just learning to (touch) type, so take it with a pinch of salt.
I have very large hands (Goes with the territory, I'm 6'3"). I still find I move off home row, but, watching my mother-oin-law (Trained to type in school, 80ish WPM), she moves from home-row far more than I do, and she has what I would call "average" hands.
Everybody has to do it. As other say, it's more about how you get back on to it.