I used my Race with reds at work for a while (until the LEDs got kinda flakey), and spend most of my day writing scripts and poking at stuff on a terminal.
I could probably do without the arrows, since I'm pretty much always in vim or using vi-mode in the shell. But, I think the ~/esc issue would bother me more. I use backticks a lot in m4, '~' interactively, and I beat the fool out of my esc key.