The big appeal I can see is resources on demand.
I was thinking about virtualizing our office. We do web development.
We have mostly single-core Sempron, Celeron, and Athlon 64 boxes. The "big boys" are a Core 2 E6400 (often broken) and an Athlon X2 4200+ which needs an OS reinstall. Most machines are 1-2G of RAM; the biggest is 3.
If we had that 48-core Opteron setup, we could set up virtual environments with a few dedicated cores per desk, and some special purpose images for Win2000, Linux, etc. for testing--if nobody was in, one person could command all 48 cores himself.
And then the lame PCs would be little more than dumb X-terminals.
I loved X terminals; our university had a big lab: 100 Win2000 PCs, 100 OS 9 Macs, 10 X terminals. I skipped the queue with my Unix skills and grabbed the terminal. I recall filling up my quota (mounted as home directory) with the Netscape cache.