Hard drive quality rant

[BucklingSpring]

In mid 80’s hard drives were outlasting the life of the computers they were in
In mid 90’s some hard drives were failing during the life of the computers
In mid 00’s many hard drives were failing during the life of the computers

Since 2010 I can’t get a Green, Red, Blue, [insert consumer] drive that holds until the warranty ends. Which is kind of a good thing, at least I can get a refurbished replacement drive that will last the half of its new counterpart.

I know I use them a lot (Virtual Machines, video rendering, big games) but jeez! It drives me nuts. Pun intended.

The only things that seem to be worth buying today are the Enterprise level drives at 2 3 times the price. Even then, I still get few problems.

Thank God  platter hard drives are more reliable than SSDs.

BTW, Western Digital just release a new Black generation (WDxxxxFZEX). The FZEX has an  improved performance and “reliability” over its 5 year warranty predecessor. Can’t wait to see how well those are performing reliability wise.

I will end this rant with words of wisdom: Backup your sh!t while you can

[127001]

Raid.

[BucklingSpring]

^Yep but... but raids when done properly only protect the data, it doesn't improve drive reliability. As a matter of fact, if you raid drives that are not built for it, you will not only get poor performance but you will also dramatically reduce its life expectation.

[The_Beast]

Raid.

Raid isn't backup......

[127001]

I understand both these things. What I'm really saying here is that the quality of drives has in no way degraded overtime. Drives have gotten orders of magnitude more complex and much much faster. There are just more things to go wrong now.

I say raid because storage is cheap now. Build in redundancy and the system is more reliable than it ever was in the 80's.

Also, I know raid is not a backup. I didn't mean for it to be. I meant it as a way of dealing with drive quality. (ps. My job is testing enterprise storage systems)

[tp4tissue]

Every time my drives make the Gurgling noise... I get chills....

I know it's just random seek... But in the back of my mind, I'm thinking... hey hard-drive slow down... don't push urself... it's ok if you get the data to me a little slower...

[DamienG]

Hard drives have always been fickle.

Most hard drives in the 80s costs thousands of dollars and your average Joe probably never used one let alone had it outlast his computer.

Drives don't need to be 'built for raid'. It's a process whereby the data is striped across the disks. They don't even know it's happening.

[)amien

[JPG]

Raids are not the answer. From personal experience, Raid =  more discs = more chance of failure =  more trouble. Yes they can be useful in critical applications (yet I have to see a Raid that does not go wrong on all disks at the same time lol), but backups are much more reliable in general. Also, having no sensible data is also much easier to manage! When your drive fails, it's a good opportunity to clean up the crap you accumulated the last 4 years! But if you have precious data, copy it (and not at the same place preferably)

[RabRhee]

Hard drives have always been fickle.

Most hard drives in the 80s costs thousands of dollars and your average Joe probably never used one let alone had it outlast his computer.

Drives don't need to be 'built for raid'. It's a process whereby the data is striped across the disks. They don't even know it's happening.

[)amien

We bought a Hewlett Packard 4x1mb hard drive in 1984 that was only $4500. We couldn't believe we could ever fill that much space.

I do remember a stack of drives failing in the 80s, mind you back then you had to manually park the heads before moving so maybe that helped the problems with people who just didn't. I also remember having to spin the head motor manually to get the damn things going pretty often on a few of our basic Tandon ATs (with Western Digital drives). failures were also very much bathtub curves, we had at least 4 fail within a few weeks of a new machine arriving.

I have only bought about 8 drives in the past 5 years and they are all going fine. most are 1-3tb WD, one 3TB samsung. A few of these are video file drives that are accessed every day by many computers, which is supposed to shorten life a great deal, but ok so far (touch wood). I would say the worst time I saw for drives was late 90s where many seemed to fail within or about a year, and were hard to return for warranty unless you had original drive boxes.

[PointyFox]

I've been using:
-4 HDDs since 2008 and have had no problems with them so far.
-1 HDD since 2006 and have had no problems with it so far.
-1 SSD since 2011 and have had no problems with it so far.

So, maybe you're just unlucky, or maybe I'm lucky?

[127001]

Raids are not the answer. From personal experience, Raid =  more discs = more chance of failure =  more trouble.

Wat?

[PointyFox]

Raids are not the answer. From personal experience, Raid =  more discs = more chance of failure =  more trouble.

Wat?

I think he's considering only data striping and not redundant disks.

[JPG]

Raids are not the answer. From personal experience, Raid =  more discs = more chance of failure =  more trouble.

Wat?

What I mean is that if you have a raid with 5 discs, each disc having the same failure rate, you then have 5 times more chance that 1 disc will fail. When it happens, if your Raid works as intended, your Raid will continue working, but you will still have the trouble of replacing you disc, so if you don't have a critical use for these discs, then it's more trouble than gain in my opinion. If you need something like 24/7, it's a different topic. But for personal use, the more discs the more chances 1 will fail, independently of the impact of such an event.

[JPG]

Raids are not the answer. From personal experience, Raid =  more discs = more chance of failure =  more trouble.

Wat?

I think he's considering only data striping and not redundant disks.

No, I was including redundant disks. If a disk fails, it still has to be replaced and still incur some trouble. And the more disks the more chance this will happen.

[127001]

Raids are not the answer. From personal experience, Raid =  more discs = more chance of failure =  more trouble.

Wat?

I think he's considering only data striping and not redundant disks.

No, I was including redundant disks. If a disk fails, it still has to be replaced and still incur some trouble. And the more disks the more chance this will happen.

Disk don't matter, data matters.

A backup is a good way to protect data incrementally. Raid protects against disk failure that can result in the loss of data not yet backed up.

Raid is VERY important.

[PointyFox]

Raids are not the answer. From personal experience, Raid =  more discs = more chance of failure =  more trouble.

Wat?

I think he's considering only data striping and not redundant disks.

No, I was including redundant disks. If a disk fails, it still has to be replaced and still incur some trouble. And the more disks the more chance this will happen.

Disk don't matter, data matters.

A backup is a good way to protect data incrementally. Raid protects against disk failure that can result in the loss of data not yet backed up.

Raid is VERY important.

..unless you're running Windows 8.  Then all you have on your computer is Candy Crush Saga or Cut the Rope.   

[JPG]

Raids are not the answer. From personal experience, Raid =  more discs = more chance of failure =  more trouble.

Wat?

I think he's considering only data striping and not redundant disks.

No, I was including redundant disks. If a disk fails, it still has to be replaced and still incur some trouble. And the more disks the more chance this will happen.

Disk don't matter, data matters.

A backup is a good way to protect data incrementally. Raid protects against disk failure that can result in the loss of data not yet backed up.

Raid is VERY important.

Depends on the context. The context will determine if they are worth it or not. For me, on a personal use, they are not. For servers, companies, data storage, etc. it can be. It's just not an "easy solve all" solution.

On my personal computer, I store nothing of importance. I can either reinstall everything, or just don't really need it. The very few data of importance that I have is stored in a mail account (my cv) and it's not really critical anyway.

For sure, if I had important/critical data on my computer, I would think about redundancy, but right now it's not a need, and it's probably not a need for many people.

[berserkfan]

I am very amused.

It is common for Americans to describe their property as shxt, but when you say that shxt needs to be backed up, it sounds as though your rig is an overstuffed toilet full of smut and crappy porn.

I will end this rant with words of wisdom: Backup your sh!t while you can

[BucklingSpring]

..unless you're running Windows 8.  Then all you have on your computer is Candy Crush Saga or Cut the Rope.   

Uh... Are you referring to cloud storage? Call me old school but I'd rather keep my data locally and know who is accessing it than dumping it on Clouds with no way to know.
Yeah yeah, they have disaster recovery plans, backup facilities, physical and environmental security policies and so forth. I'm just not ready for it. Plus even with all the BS they use to sell it, it still fail every one and then.

[BucklingSpring]

I am very amused.

It is common for Americans to describe their property as shxt, but when you say that shxt needs to be backed up, it sounds as though your rig is an overstuffed toilet full of smut and crappy porn.

I will end this rant with words of wisdom: Backup your sh!t while you can

Holly f*ck... How did you know?
Are you sysadmin  at my Cloud provider?

[TheSoulhunter]

Out of the 28 HDDs I used in the last 10 years only 2 failed...
All are/were used for 3-5 years, atm I run 11 HDDs, no failure the last 3 years!

[eth0s]

I've been using:
-4 HDDs since 2008 and have had no problems with them so far.
-1 HDD since 2006 and have had no problems with it so far.
-1 SSD since 2011 and have had no problems with it so far.

So, maybe you're just unlucky, or maybe I'm lucky?

If you have never had an HDD or SSD fail ever in your life, then you have been incredibly lucky. 

I am very amused.

It is common for Americans to describe their property as shxt, but when you say that shxt needs to be backed up, it sounds as though your rig is an overstuffed toilet full of smut and crappy porn.

Hmm, what are you talking about?   I am an American, and I have never described my "property as shxt", nor have any of the other Americans I know.  Also, who wastes time and money backing up pr0n?  I think you are projecting your own feelings more than you know.

Disk don't matter, data matters.

A backup is a good way to protect data incrementally. Raid protects against disk failure that can result in the loss of data not yet backed up.

Raid is VERY important.

^+1.  This is absolutely correct, and I can't say it any better.

What I mean is that if you have a raid with 5 discs, each disc having the same failure rate, you then have 5 times more chance that 1 disc will fail. When it happens, if your Raid works as intended, your Raid will continue working, but you will still have the trouble of replacing you disc, so if you don't have a critical use for these discs, then it's more trouble than gain in my opinion. If you need something like 24/7, it's a different topic. But for personal use, the more discs the more chances 1 will fail, independently of the impact of such an event.

This is totally wrong.  This is not how probability works.  The chance of failure for each HDD remains the same, no matter how many HDD's you have.  There is no aggregation of failure rate in building a RAID 1 or RAID 5 array.  (You will however increase your failure rate by making a RAID 0 array.)   

The failure rate of a 3-year old HDD is approximately 8.7%.  Which means that you have a 87/1000 chance of an old HDD failing at any time the HDD spins.   If you put two (2) old HDD's in a RAID 1 array, the chance that any one HDD will fail is still only 87/1000.  However, the chance that BOTH will fail, and that you will lose everything, is reduced to .087 x .087 = .0076, which is a 0.76% chance of losing everything.  Meaning that you can go from an 8.7% chance of losing everything down to only a 0.76% chance of losing everything, by putting two HDD's in RAID 1.  And if you use 5 redundant disks in your RAID array, your chance goes down to .086^5 = .00005% of losing everything. 

However, I agree with you that RAID is complicated, expensive and sometimes difficult to set up and maintain, due to software limitations.  And, I take your point that in terms of practical use, RAID is still a PITA.  However, in terms of pure mathematical probability, it is far superior.

[jwaz]

http://en.wikipedia.org/wiki/RAID

Clearly there are a few of you who could use a thorough read of this.

[MTManiac]

you get what you pay for, HD's are no different

IMHO it's easier to setup a RAID to ensure your data is safe from hardware failure than it is to back it up constantly

[Thimplum]

RAID haters totally fail, guys!

fixed

[tp4tissue]

you get what you pay for, HD's are no different

IMHO it's easier to setup a RAID to ensure your data is safe from hardware failure than it is to back it up constantly

^^^ LOL, raid noobs...

raid =\= back up

[JPG]

I've been using:
-4 HDDs since 2008 and have had no problems with them so far.
-1 HDD since 2006 and have had no problems with it so far.
-1 SSD since 2011 and have had no problems with it so far.

So, maybe you're just unlucky, or maybe I'm lucky?

If you have never had an HDD or SSD fail ever in your life, then you have been incredibly lucky. 

I am very amused.

It is common for Americans to describe their property as shxt, but when you say that shxt needs to be backed up, it sounds as though your rig is an overstuffed toilet full of smut and crappy porn.

Hmm, what are you talking about?   I am an American, and I have never described my "property as shxt", nor have any of the other Americans I know.  Also, who wastes time and money backing up pr0n?  I think you are projecting your own feelings more than you know.

Disk don't matter, data matters.

A backup is a good way to protect data incrementally. Raid protects against disk failure that can result in the loss of data not yet backed up.

Raid is VERY important.

^+1.  This is absolutely correct, and I can't say it any better.

What I mean is that if you have a raid with 5 discs, each disc having the same failure rate, you then have 5 times more chance that 1 disc will fail. When it happens, if your Raid works as intended, your Raid will continue working, but you will still have the trouble of replacing you disc, so if you don't have a critical use for these discs, then it's more trouble than gain in my opinion. If you need something like 24/7, it's a different topic. But for personal use, the more discs the more chances 1 will fail, independently of the impact of such an event.

This is totally wrong.  This is not how probability works.  The chance of failure for each HDD remains the same, no matter how many HDD's you have.  There is no aggregation of failure rate in building a RAID 1 or RAID 5 array.  (You will however increase your failure rate by making a RAID 0 array.)   

The failure rate of a 3-year old HDD is approximately 8.7%.  Which means that you have a 87/1000 chance of an old HDD failing at any time the HDD spins.   If you put two (2) old HDD's in a RAID 1 array, the chance that any one HDD will fail is still only 87/1000.  However, the chance that BOTH will fail, and that you will lose everything, is reduced to .087 x .087 = .0076, which is a 0.76% chance of losing everything.  Meaning that you can go from an 8.7% chance of losing everything down to only a 0.76% chance of losing everything, by putting two HDD's in RAID 1.  And if you use 5 redundant disks in your RAID array, your chance goes down to .086^5 = .00005% of losing everything. 

However, I agree with you that RAID is complicated, expensive and sometimes difficult to set up and maintain, due to software limitations.  And, I take your point that in terms of practical use, RAID is still a PITA.  However, in terms of pure mathematical probability, it is far superior.

I am not saying that Raid is not good, but if you have 5 drives, all 5 drives can fail, so it's 5 time more chances than 1 drive, so if you don't need the increase availability of a Raid, then don't go for it. If you need it, then do it right. All I wanted to say.

[TacticalCoder]

The failure rate of a 3-year old HDD is approximately 8.7%.  Which means that you have a 87/1000 chance of an old HDD failing at any time the HDD spins.

(I'm not into the RAID discussion at all but...)

"at any time the HDD spins"!?

Aren't you talking about the "Annualized Failure Rate" instead?

In this Wikipedia article:

http://en.wikipedia.org/wiki/Annualized_failure_rate

it is written that:

"Google's 2007 study found, based on a large field sample of drives, that actual AFRs for individual drives ranged from 1.7% for first year drives to over 8.6% for three-year old drives."

If there was really a 8.7% chance that any 3-years old+ drive would die every time it spins (instead of an 8.7% AFR), I'm pretty sure I would have had more than three or four of my personal hard drives dying since the late eighties 

[webs0r]

Just bringing back to the OP, totally agree.

Hope someone magically comes up with a 100% reliable mass storage device not too far in the future so we don't have to worry about this anymore.

I've got a non-mission-critical file server with 11 disks (slowly growing) that has been running 24/7. I've had 2 disks fail on me.
I haven't lost any data. Disks in there range from about 5 (maybe 6?) years old to now.

It is a pain though to have a strategy, set it up and maintain it to be able to ensure your data is safe. It's just too much of a PITA.
An easier way to do it would be nice.

Curious about what solutions you guys use?

I've been using FlexRAID for quite some years now.
I know similar solutions like Drivebender and some other one, I forget the name, have come out more recently. Anyone have nice easy hassle free experiences with them they want to share?
Curious about software redundancy, hardware redundancy, backup solutions you use.

So I don't have striped data. Lose 1 disk, and I would only be exposed to risk of losing content on that disk.
I don't have a full backup of the mass of data (I dunno about ~18TB).
I keep a cold spare available to minimise data unavailability.
Similar to above, if a disk goes down only the contents of that disk is unavailable. But I have to ensure no one writes or deletes existing data on the "array" when an incident occurs, otherwise I am exposed to increased data loss.

[BucklingSpring]

"Google's 2007 study found, based on a large field sample of drives, that actual AFRs for individual drives ranged from 1.7% for first year drives to over 8.6% for three-year old drives."
If there was really a 8.7% chance that any 3-years old+ drive would die every time it spins (instead of an 8.7% AFR), I'm pretty sure I would have had more than three or four of my personal hard drives dying since the late eighties 

I know Google must have a load of harddrives in their data centers. I wonder if the study was based on all kinds of hard drives or "Enterprise" grade drives.

What I would really like to see and it will never be is stats on the SMART data for defective drives returned under warranty. "Average number of ON/OFF cycles and hours of use)

AFR is one thing... I used to buy drive with high advertised MTBF (Mean Time Before Failure) and higher warranty period.

When Western Digital was still advertising MTBFs the Enterprise drives were ~2-3x the consumer's

Now they no longer advertise it and all their drives (Consumer and Enterprise) has "Non-recoverable read errors per bits read < 1 in 10exp14"
All the fking same. Except for warranty period there is absolutely no advertised data to backup reliability (what drive is more reliable than others)

WD REx, Velociraptor and Black still got a 5 years warranty.

I think the SSD era with its continuous reliability improvements and decreasing price is about to kick the platters in the face.

[BucklingSpring]

I don't mean to derail but more or less related to the OP is the lifespan of Digital Data vs paper.

A quote from the Net (Not sure how reliable it is but just for sake of argument)

"Media Life Expectancy

All digital media has finite life spans which are dependent on a number of factors, including manufacturing quality, age and condition before recording, handling and maintenance, frequency of access, and storage conditions.  Studies have indicated that under optimal conditions, the life expectancy of magnetic media ranges from 10 to 20 years for different types, while optical media may last as long as 30 years.  However, in real life situations, most media life expectancies are significantly less."

NASA has rooms filled of old original film footages turning into dust.

I found very interesting that today our most valuable asset is knowledge and information. Surprisingly enough, we are currently storing it on the most volatile media ever (after voice).

In thousands of years, I don't think the aliens will be able to find any traces of Ripster's $1000 reddit on the face of the planet.
They will never be able to see how evolved our current civilization was.

Shame on short life media!

[DamienG]

Reminds me of the BBC Domesday project revival.

Basically almost thousand years ago (1086) there was a big census of life in the England put into a big book called Domesday book. It's still readable and provides interesting insights into 11th century life there.

In the 80s the BBC did a similar thing with video, text and audio to commemorate the 900th anniversary about 80s life in the UK and stored it on laser discs connected to a popular British computer used in schools made by a company called Acorn.

Within 20 years it was incredibly hard to access. It's basically taken them a year to reverse engineer and extract the data back off

[)amien

[PointyFox]

Reminds me of the BBC Domesday project revival.

Basically almost thousand years ago (1086) there was a big census of life in the England put into a big book called Domesday book. It's still readable and provides interesting insights into 11th century life there.

In the 80s the BBC did a similar thing with video, text and audio to commemorate the 900th anniversary about 80s life in the UK and stored it on laser discs connected to a popular British computer used in schools made by a company called Acorn.

Within 20 years it was incredibly hard to access. It's basically taken them a year to reverse engineer and extract the data back off

[)amien

 

[BucklingSpring]

Right that is another extremely valid point. If you have the most reliable and durable media in the world but the device to read it no longer exists... Then who cares about reliability.

That's a good thing about stuff we can read just by looking at it. After so many years, the only challenge is "Translation". As for digital media. You first need to reverse engineer the media, then try to interpret what's on it. I wonder how big of a paper stack would be needed to print 4TB worth of Text.

Reminds me of the BBC Domesday project revival.

Basically almost thousand years ago (1086) there was a big census of life in the England put into a big book called Domesday book. It's still readable and provides interesting insights into 11th century life there.

In the 80s the BBC did a similar thing with video, text and audio to commemorate the 900th anniversary about 80s life in the UK and stored it on laser discs connected to a popular British computer used in schools made by a company called Acorn.

Within 20 years it was incredibly hard to access. It's basically taken them a year to reverse engineer and extract the data back off

[)amien

 

[audioave10]

I've had good luck with WD blues & blacks...NO failures. I buy from Newegg and live close to one of their warehouses so I don't seem to get shipping damage ever. Shipping & handling is part of the problem here I think. Check the many videos of a UPS shipping hub in action...they literally throw packages around.

[rowdy]

I've had good luck with WD blues & blacks...NO failures. I buy from Newegg and live close to one of their warehouses so I don't seem to get shipping damage ever. Shipping & handling is part of the problem here I think. Check the many videos of a UPS shipping hub in action...they literally throw packages around.

Why I prefer to pickup HDDs in person.  Hoping that they were delivered gently to the store.

[BucklingSpring]

Yet another very good point.

Once upon a time, drives were boxed. Now in the store they throw you the OEM's in the sealed magnetic protection bag. That's it.
So how did the kid shook them and dropped them on the shelf? The only reassuring point here is HD may no longer have advertised MTBF but they do have shock's and vibration's. Those number are very high comparing to older drives.

Why I prefer to pickup HDDs in person.  Hoping that they were delivered gently to the store.

[PointyFox]

I've had good luck with WD blues & blacks...NO failures. I buy from Newegg and live close to one of their warehouses so I don't seem to get shipping damage ever. Shipping & handling is part of the problem here I think. Check the many videos of a UPS shipping hub in action...they literally throw packages around.

I opened my front door once to see a package hitting the outer screen door.  I stepped outside and the UPS guy was in the street, stepping back into his truck. Apparently he tossed it at my door from 30 feet away. 

[dorkvader]

Just print out your important data onto paper as QR codes (or other). If you use the right ink, it'll even be waterproof and also won't fade.

I don't use RAID, but I keep my data on drives that are successively backed up. So every week, I transfer all important stuff from my main drive (SSD) to my first backup drive and spin it down to put back in the corner. Every few weeks, I take that drive and back up the new stuff to the pair of main backup drives that are elsewhere. I'm not paranoid about my data, just lazy.

After having lost data once, in 2008, I learned my lesson and things I really care about (original artwork, papers, writings, etc) I back up everywhere I can, online, offline, flash drives, SSD. I have some documents on all 12 of my hard drives, and on about 3-5 of my "storage" flashdrives as well as various places online. Not only does it make it less likely to be lost again, it's also easier to find.

I agree with some people: hard drives nowadays are very finely balanced, to get them so fast and have the data density so high. To be honest, I'm surprised they don't have a higher read error rate than they do. Back in the day they used good components to make drives, but there was a lot more room for error, since the bits took up larger spaces on the platter. I mean, they got away with stepper motors for the read head instead of voice coils!

RAID is pretty useful, statistics are important, and while Google's survey is an interesting read, I wonder of it's utility, so many years hence.