So I've had an iPhone 5 for two weeks now, here are my complaints, btw I was on an android device the two years prior to this (thank god i'm done with that o.o)
First off. 'Scuffgate' I'd recommend staying away from the Black iP5, even though IMO it looks waaaay better than it's white counterpart, it will scuff, and it will look pretty bad after a year or so, attacked is a picture of my headphone jack, which I've been using pretty regularly every day.

Second. iOS is a bit annoying, I really dislike how I can't easily get the date in the notification center, and since the iP5 hasn't been jailbroken yet, I'm stuck waiting for apple to update iOS, or wait for jailbreak to add this functionality. This is more of a personal preference, but it's something I really miss from Android.
If you're upgrading from a previous iPhone, the bigger screen will take a little while to get used to, I'm still working towards being comfortable using the larger screen. Again personal preference, and minor nitpick.
Last thing I can think of bad, is the camera "purple tint" when you take a picture with a light source just out of range, but from what I'm hearing this is occurring on the new Android phones too, so it's not Apple-specific.
Onto the good! I know you asked for reasons not to buy it, but I gotta list good with the bad

This is hands down, the best hardware on the market right now, the plastic GSIII cannot begin to touch this in hardware feel and raw performance, Apple has done some magic with the A6 processor.
The backing on the phone is also really ingenius, at first I thought it was pretty stupid to be 3/4 aluminum with the two strips of glass, but after my two weeks, I've realized the purpose of the glass is to be able to pull your phone out of your pocket easier. Since your oily fingers slip on metal, and grip the glass, it's easy to pull your phone out no matter which way it fits in your pocket.
This thing is thin. Ridiculously thin and light, I had doubt about it's batter life, but with my daily usage of 5-8h of web-browsing over wifi, 2h of Bluetooth audio, and the remainder of the day polling email servers every 15 minutes, I can get a whole day+ of battery life (6AM-6PM and I'm at 30% right now)
Final thing I can think of, is the Lightning cable, this was a MUCH needed change to the Apple docking/charging cable, the old plug is so big, it's crazy it took them this long to release this, and the whole "fits in either way you plug it in" is absolutely brilliant, just grab the cable and stick it in, no need to fiddle around with it.
These are just some of my thoughts of the iPhone after like I said, 2 weeks of use.
If I had to choose again, I'd probably grab the white iP5 instead, because I like naked phones, so this poor black one will have quite the tough life.