When No Redundancy Is More Reliable – The Myth of Redundancy

Risk in a difficult concept and it requires a lot of training, thought and analysis to properly assess given scenarios.  Often, because risk assessments are so difficult, we substitute risk analysis with simply adding basic redundancy and assuming that we have appropriately mitigated risk.  But very often this is not the case.  The introduction of complexity or additional failure modes often accompany the addition of redundancy and these new forms of failure have the potential to add more risk than the added redundancy removes.  Storage systems are especially prone to these decision processes which is unfortunate as few, if any, systems are so susceptible to failure and more important to protect.

RAID is a great example of where a lack of holistic risk thinking can lead to some strange decision making.  If we look at a not uncommon scenario we will see where the goal of protecting against drive failure can actually lead to an increase in risk even when additional redundancy is applied.  In this scenario we will compare a twelve drive array consisting of twelve three terabyte SATA hard drives in a single array.  It is not uncommon to hear of people choosing RAID 5 for this scenario to get “maximum capacity and performance” while having “adequate protection against failure.”

The idea here is that RAID 5 protects against the loss of a single drive which can be replaced and the array will rebuild itself before a second drive fails.  That is great in theory, but the real risks of an array of this size, thirty six terabytes of drive capacity, come not from multiple drive failures as people generally suspect but from an inability to reliably rebuild the array after a single drive failure or from a failure of the array itself with no individual drives failing.  The risk of a second drive failing is low, not non-existent, but quite low.  Drives today are highly reliable. Once one drives fails it does increase the likelihood of a second drive failing, which is well documented, but I don’t want this risk to mislead us from looking at the true risks – the risk of a failed resilvering operation.

What happens that scares us during a RAID 5 resilver operation is that an unrecoverable read error (URE) can occur.  When it does the resilver operation halts and the array is left in a useless state – all data on the array is lost.  On common SATA drives the rate of URE is 10^14, or once every twelve terabytes of read operations.  That means that a six terabyte array being resilvered has a roughly fifty percent chance of hitting a URE and failing.  Fifty percent chance of failure is insanely high.  Imagine if your car had a fifty percent chance of the wheels falling off every time that you drove it.  So with a small (by today’s standards) six terabyte RAID 5 array using 10^14 URE SATA drives, if we were to lose a single drive, we have only a fifty percent chance that the array will recover assuming the drive is replaced immediately.  That doesn’t include the risk of a second drive failing, only the risk of a URE failure.  It also assumes that the drive is completely idle other than the resilver operation.  If the drives are busily being used for other tasks at the same time then the chances of something bad happening, either a URE or a second drive failure, begin to increase dramatically.

With a twelve terabyte array the chances of complete data loss during a resilver operation begin to approach one hundred percent – meaning that RAID 5 has no functionality whatsoever in that case.  There is always a chance of survival, but it is very low.  At six terabytes you can compare a resilver operation to a game of Russian roulette with one bullet and six chambers and you have to pull the trigger three times.  With twelve terabytes you have to pull it six times!  Those are not good odds.

But we are not talking about a twelve terabyte array.  We are talking about a thirty six terabyte array – which sounds large but this is a size that someone could easily have at home today, let alone in a business.  Every major server manufacturer, as well as nearly all low cost storage vendors, make sub $10,000 storage systems in this capacity range today.  Resilvering a RAID 5 array with a single drive failure on a thirty six terabyte array is like playing Russian roulette, one bullet, six chambers and pulling the trigger eighteen times!  Your data doesn’t stand much of a chance.  Add to that the incredible amount of time needed to resilver an array of that size and the risk of a second disk failing during that resilver window starts to become a rather significant threat.  I’ve seen estimates of resilver times climbing into weeks or months on some systems.  That is a long time to run without being able to lose another drive.  When we are talking hours or days the risks are pretty low, but still present.  When we are talking weeks or months of continuous abuse, as resilver operations are extremely drive intensive, the failure rates climb dramatically.

With an array of this size we can effectively assume that the loss of a single drive means the loss of the complete array leaving us with no drive failure protection at all.  Now if we look at a drive of the same or better performance with the same or better capacity under RAID 0, which also has no protection against drive loss, we need only use eleven of the same drives that we needed twelve of for our RAID 5 array.  What this means is that instead of twelve hard drives, each of which has a roughly three percent chance of annual failure, we have only eleven.  That alone makes our RAID 0 array more reliable as there are fewer drives to fail.  Not only do we have fewer drives but there is no need to write the parity block nor skip parity blocks when reading back lowering, ever so slightly, the mechanical wear and tear on the RAID 0 array for the same utilization giving it a very slight additional reliability edge.  The RAID 0 array of eleven drives will be identical in capacity to the twelve drive RAID 5 array but will have slightly better throughput and latency.  A win all around.  Plus the cost savings of not needing an additional drive.

So what we see here is that in large arrays (large in capacity, not in spindle count) that RAID 0 actually passes RAID 5 in certain scenarios.  When using common SATA drives this happens at capacities experienced even by power users at home and by many small businesses.  If we move to enterprise SATA drives or SAS drives then the capacity number where this occurs becomes very high and is not a concern today but will be in just a few years when drive capacities get larger still.  But this highlights how dangerous RAID 5 is in the sizes that we see today.  Everyone understands the incredible risks of RAID 0 but it can be difficult to put into perspective that RAID 5’s issues are so extreme that it might actually be less reliable than RAID 0.

That RAID 5 might be less reliable than RAID 0 in an array of this size based on resilver operations alone is just the beginning.  In a massive array like this the resilver time can take so long and exact such a toll on the drives that second drive failure starts to become a measurable risk as well.  And then there are additional risks caused by array controller errors that can utilize resilver algorithms to destroy an entire array even when no drive failure has occurred.  As RAID 0 (or RAID 1 or RAID 10) do not have resilver algorithms they do not suffer this additional risk.  These are hard risks to quantify but what is important is that they are additional risks that accumulate when using a more complex system when a simpler system, without the redundancy, was more reliable from the outset.

Now that we have established that RAID 5 can be less reliable than RAID 0 I will point out the obvious dangers of RAID 0.  RAID in general is used to mitigate the risk of a single, lone hard drive failing.  We all fear a single drive simply failing and all data being lost.  RAID 0, being a large stripe of drives without any form of redundancy, takes the risk of data loss of a single drive failing and multiplies it across a number of drives where any drive failing causes total loss of data to all drives.  So in our eleven disk example above, if any of the eleven disks fails all is lost.  It is clear to see where this is dramatically more dangerous than just using a single drive, all alone.

What I am trying to point out here is that redundancy does not mean reliability.  Just because something is redundant, like RAID 5, provides no guarantee that it will always be more reliable than something that is not redundant.

My favourite analogy here is to look at houses in a tornado.  In one scenario we build a house of brick and mortar.  On the second scenario we build two redundant house, east out of straw (our builders are pigs, apparently.)  When the tornado (or big bad wolf) comes along which is more likely to leave us with a standing house?  Clearing one brick and mortar house has some significant reliability advantages over redundant straw houses.  Redundancy didn’t matter, reliability mattered in the end.

Redundancy is often misleading because it is easy to quantify but hard to qualify.  Redundancy is a black or white question: Is it redundant?  Yes or no.  Simple.  Reliability is not so simple.  Reliability is about failure rates and likelihoods.  It is about statistics and analysis.  As it is hard to quantify reliability in a meaningful way, especially when selling a project to the business people, redundancy often becomes a simple substitute for this complex concept.

The concept of using redundancy to misdirect questions of reliability also ends up applying to subsystems in very convoluted ways.  Instead of making a “system” redundant it has become common to make a highly reliable, and low cost, subsystem redundant and treat subsystem redundancy as applying to the whole system.  The most common example of this is RAID controllers in SAN products.  Rather than having a redundant SAN (meaning two SANs) manufacturers will often make that one component not often redundant in normal servers redundant  and then calling the SAN redundant – meaning a SAN that contains redundancy, which is not at all the same thing.

A good analogy here would be to compare having redundant cars meaning two complete, working cars and having a single car with a spare water pump in the trunk in case the main one fails.  Clearly, a spare water pump is not a bad thing.  But it is also a trivial amount of protection against car failure compared to having a second car ready to go.  In one case the entire system is redundant, including the chassis.  In the other we are making just one, highly reliable component redundant inside the chassis.  It’s not even on par with having a spare tire which, at least, is a car component with a higher likelihood of failure.

Just like the myth of RAID 5 reliability and system/subsystem reliability, shared storage technologies like SANs and NAS often get treated in the same way, especially in regards to virtualization.  There is a common scenario where a virtualization project is undertaken and people instinctively panic because a single virtualization host represents a single point of failure where, if it fails, many systems will all fail at once.

Using the term “single point of failure” causes a panic feeling and is a great means of steering a conversation.  But a SPOF, as we like to call it, while something we like to remove when possible may not be the end of the world.  Think about our brick house.  It is a SPOF.  Our two houses of straw are not.  Yet a single breeze takes out our redundant solutions faster than our reliable SPOF.  Looking for SPOFs is a great way to find points of fragility in a system, but do not feel that every SPOF must be made redundant in every scenario.  Most businesses will find their best value having many SPOFs in place.  Our real goal is reliability at appropriate cost, redundancy, as we have seen, is no substitute for reliability, it is simply a tool that we can use to achieve reliability.

The theory that many people follow when virtualizing is that they take their virtualization host and say “This host is a SPOF, so I need to have two of them and use High Availability features to allow for transparent failover!”  This is spurred by the leading virtualization vendor making their money firstly by selling expensive HA add on products and secondly by being owned by a large storage vendor – so selling unnecessary or even dangerous additional shared storage is a big monetary win for them and could easily be the reason that they have championed the virtualization space from the beginning.  Redundant virtualization hosts with shared storage sounds great but can be extremely misguided for several reasons.

The first reason is that removing the initial SPOF, the virtualization host, is replaced with a new SPOF, the shared storage.  This accomplishes nothing.  Assuming that we are using comparable quality servers and shared storage all we’ve done is move where the risk is, not change how big it is.  The likelihood of the storage system failing is roughly equal to the likelihood of the original server failing.  But in addition to shuffling the SPOF around like in a shell game we’ve also done something far, far worse – we have introduced chained or cascading failure dependencies.

In our original scenario we had a single server.  If the server stayed working we are good, if it failed we were not.  Simple.  Now we have two virtualization hosts, a single storage server (SAN, NAS, whatever) and a network connecting them together.  We have already determined that the risk of the shared storage failing is approximately equal to our total system risk in the original scenario.  But now we have the additional dependencies of the network and the two front end virtualization nodes.  Each of these components is more reliable than the fragile shared storage (anything with mechanical drives is going to be fragile) but that they are lower risk is not the issue, the issue is that the risks are combinatorial.

If any of these three components (storage, network or the front end nodes) fail then everything fails.  The solution to this is to make the shared storage redundant on its own and to make the network redundant on its own.  With enough work we can overcome the fragility and risk that we introduced by adding shared storage but the shared storage on its own is not a form of risk mitigation but is a risk itself which must be mitigated.  The spiral of complexity begins and the cost associated with bringing this new system up on par with the reliability of the original, single server system can be astronomic.

Now that we have all of this redundancy we have one more risk to worry about.  Managing all of this redundancy, all of these moving parts, requires a lot more knowledge, skill and preparation than does managing a simple, single server.  We have moved from a simple solution to a very complex one.  In my own anecdotal experience the real dangers of solutions like this come not from the hardware failing but from human error.  Not only has little been done to avoid human error causing this new system to fail but we’ve added countless points where a human might accidentally bring the entire system, redundancy and all, right down.  I’ve seen it first hand; I’ve heard the horror stories.  The more complex the system the more likely a human is going to accidentally break everything.

It is critical that as IT professionals that we step back and look at complete systems and consider reliability and risk and think of redundancy simply as a tool to use in the pursuit of reliability.  Redundancy itself is not a panacea.  Neither is simplicity.  Reliability is a complex problem to tackle.  Avoiding simplistic replacements is an important first step in moving from covering up reliability issues to facing and solving them.


16 thoughts on “When No Redundancy Is More Reliable – The Myth of Redundancy”

  1. SAM, I enjoyed this article quite a lot. Good after readying the OpenStorage article.
    I have been gathering, readying a lot of information, and in the end (a couple of weeks ago) I finally decided that the best route was to use internal datastore in our server because of higher I/O available, and more reliable drives than our NAS, but replicate data to NAS, in case that there is a problem with the local store. Since I already have to VM Host (to separate resources consumption) this seemed more logical.
    The weird thing, is that I did not see much about doing this kind of deployment, and I talked with a couple of people that put a lot of emphasis into Vmotion and HA, however, they do not have large budgets either. I was really feeling lonely 🙂
    However, the challenge is still there to have this replication.
    I found this software
    That can present local datastore as a SAN (using iSCSI), but I am still wondering if there are other options out there

  2. Fantastic write up. I saw first hand the exact scenario you laid out in the last part of your article, so I have seen the danger up jumping on the “Lets get a SAN” bandwagon.

  3. Nice write up. When using than 6 or more drives, I have never considered a RAID5 as a ‘safe’ method of redundancy. How does using a RAID6 compare against the shortfalls you covered of RAID5 rebuilding a degraded array?

  4. There are several problems with the indictment of RAID5.

    First, the problems identified are more related to giant drives and RAID volumes that span a number of them than specifically to RAID5. The same URE rates apply to large volumes regardless of underlying RAID. The risks outlined are mitigated by using giant drives in sets rather than all in one giant RAID. For example, the 12-drive RAID5 array of giant disks should be set up as three RAID5 arrays, each smaller and less risky to resilver. There is a capacity loss versus using one giant RAID5, but as you have pointed out, giant RAID volumes are not viable anywa.

    Second, the brick house/straw house analogy is flawed. It is not a hurricane that takes out the first member of a redundant pair. Unlike a hurricane, the root cause of the system (brick house) that fails is not so likely to take out the straw house as a “hurricane.” This makes the redundant second system, even built of straw, better than indicated in the article.

    Third, the car analogy is problematic. Systems with redundancy built in is not like carrying a spare in the truck, like a tire or a water pump. A fault tolerant car with redundant systems has two working water pumps installed and can operate when one dies. It has more than four tires, such that one or more can go flat and the vehicle still operate. Same with battery, a second one that keeps the car working when one fails.

    Finally, the article says that redundancy is itself not a fix for risk, but then says that the fix is redundancy in all components (server, san, AND network). It is clear that redundancy is the best mitigation for the risk of failure.

    In summary, the take-ways from this article:
    1. do not use giant RAID volumes
    2. when building redundant systems, make sure to include components that are often over-looked, like network.

  5. A great write-up, really enjoy reading it.


    With a 4-drive RAID 5 you still get a very large volume. Even a 3TB array gives you 25% possibility of encountering a URE during the rebuild process. I certainly won’t risk it, not in business world.

  6. Sir, you could not be more right. Thanks for “expressing” this, I definitely agree.

  7. I’m surprised how many people here are still talking about using hardware Raid for critical data. To be very frank, the idea of using ANY hardware raid configuration, (including the popular 1 & 5/6 configuration) is bordering on antiquarian. It may be time to re-evaluate. You may be “renting DVD’s” when the rest of world is streaming videos… On the beach… with an iPad-Super-Ultra-Air².

    If your data is important to you, you should be storing it in a ZFS array. If your data matters, and you don’t think you need ZFS, you simply haven’t thought about it long enough. We’ve grown accustomed to our processors and memory evolving like crazy. Now it’s time for the storage paradigm to do the same.

    With ZFS:
    -Problems like “Bit rot”, write holes, data corruption… all disappear.
    -You can create historic point-in-time snapshots (that can be re-mounted ‘read/write’ in parallel with your current data. Did a virus just destroy all of your data. No problem. Just go back in time to a few moments before the event.
    -Checksums are so ample that the chance of undetected data corruption is effectively zero, even if your dataset has more bits than there are atoms in the universe.
    -Monthly scrubs run on schedule to true-up data to its checksums. If data has changed due to bit rot, it can be corrected by a redundant copy.
    -Copy on write architecture ensures that all changes to your data are written and validated before the previous version is ‘decoupled’ from the file name.
    -Did the power go out right in the middle of writing to a critical file? No problem. The file is not corrupted because the previous state is still intact.
    -Abstraction of physical storage pools to the logical datasets contained within, so different data can have appropriate snapshotting, quota, compression, and replication policies applied.
    -You can build as much redundancy as you like. You can create as many mirrored copies as you want (to use the author’s ‘water pump’ example), but better still, you can also replicate the entire file system to other instances on or off site, through a TCP socket or SSH tunnel. Following copies receive changed data blocks, not whole files, so data transfer is manageable. If you’re ultra-paranoid like me, you can use good old-fashioned RSync to another following server for mega-ultra redundancy by way of replicating with a fully dissimilar technology.

    Oh yeah, did I mention this is all free? Sorry. I meant to.
    Granted, ZFS on Solaris (or Solaris forks), on *BSD can be a bit intimidating, but when ZFS is packaged in GUI-driven Open-source distributions like FreeNAS, or paid, supported ZFS products (like NexentaStore, and TrueNAS), it becomes something that ‘normal’ admins (like me) can incorporate into enterprise (and personal) workflows. Even my personal windows workstation libraries replicate to ZFS stores which in turn replicate offsite within minutes. House burned down? Oh well.

    EVEN IF you ignore the mind-bending read performance that can be achieved with ZFS through large read caches (DRAM, and L2 Adaptive Read Cache), and EVEN IF you ignore the insane synchronous write performance when paired with a high performance write cache device; the data integrity achievable with ZFS instance is unmatched. You owe it to yourself, to your career, perhaps to your employer to explore this technology.

    Just sayin…


  8. I agree with Mr. Karl. Me thinks a straw man is taking a beating here.

    Way back when, many of we sysops felt that RAID-5 was a way to sell more disk drives to companies. Most real configurations were simply mirrored. And we also backed up to tape–and most importantly TESTED tape restores.

    When we swapped out a failed drive in a mirror, it was always a crap-shoot. We knew it could fail–and hey, that’s why we had the tapes. But if it *didn’t* fail, bonus! We got a key piece of infrastructure up and running before midnight.

    Redundancy should be performed. Whether disk-to-disk backup, co-location, streaming tape, streaming replication…flash drives, whatever. RAID5 may not guarantee no downtime or data loss, but it’s only a layer in the sandwich of disaster recovery strategies. Then again, who (other than salesman) said RAID5 guarantee these things without fail? If your server has a single primary fixed disk, you are dead in the water. With RAID5, there’s a good possibility that there’s still time to keep running, pull critical data, et cetera–depending on configuration and circumstances.

    Unfortunately this article will be taken the wrong way by businesses, and since Statistics is (unbelievably!) still an optional math discipline in public schools, I can pretty much guarantee that the end result will be a lopsided risk plan where saving a few grand by avoiding a passable DR system will be more important than protection millions worth of data. And that’s just stupid.

    That’s just the business perspective. Every day, millions of photographs, hours of audio, and pages of writing and stuffed into systems with no redundancy and not guarantee of serviceability. Data is lost, careers are ruined, and we all pay down the debt of poor disaster recovery planning with our wages and the prices we pay at the register.

  9. I’m not deeply involved in IT (just a photographer with reasonable storage needs) and came across this article.

    I agree wholeheartedly with the main point, which I believe is for people not to confuse redundancy, with reliability, with security of data. However, this point is somewhat obscured by the completely erroneous mathematics! If it were correct, no moderately large Raid 5 array would ever be recovered. Never. Never, ever. I’m pretty sure, even with my limited experience that is wrong.

    Indeed, I’ve seen it done. Still, I do entirely agree with the position of not relying on half-baked redundancy ideas for long-term data maintenance and those who do deserve the consequences.

  10. No, even very large RAID 5 arrays would still recover once in a while. But few large ones are. It does happen, but mostly in contrived tests that don’t reflect the math because of this effect: https://mangolassi.it/topic/5084/why-non-uniform-ure-distribution-may-make-parity-raid-riskier-than-thought

    But keep in mind that effectively no large array is on RAID 5 and hasn’t been for a long time, large arrays are normally on SAS making RAID 5 overly expensive rather than “always going to fail” due to paying for lower URE failure rates and no matter how risky the math appears, it never hits 100% fail (nor 100% recovery.) So we will see large arrays occasionally recover, just as we commonly see tiny ones fail to recover even though the chances of recovery are very high.

  11. I’m sorry but your whole conjecture is ridiculous. I know this is an old post but in the case that someone stumbles across it again (like I have), I need to have a say here to correct some serious inaccuracies.

    Firstly, though, I will say that the general message is definitely valid – as re-iterated by even the non-IT guy:
    “I do entirely agree with the position of not relying on half-baked redundancy ideas for long-term data maintenance and those who do deserve the consequences.”

    To the point in question, however.

    You want to argue that R5 is –less– resilient than R0. But your whole case is based on what happens –AFTER– the first drive failure. Now, no matter –what– the chances of the R5 successfully rebuilding are, there is a still –SOME– chance. Whereas with R0, your data is already gone, 100%, with the first drive loss.

    The only way that R5 could be -less- resilient than R0, is if you could show that the very fact of creating an array with write+parity write operations statistically increases the risk of an –initial– drive failure, such that the chance of the initial failure AND that of the secondary URE during a rebuild, together, increases past the likelihood of single disk failure in a R0. And this is clearly never going to be the case. Well, not unless some ridiculously poorly-written RAID contoller software does some stupid/insane activity with the disks while in the array.

    As for all these reports that “no-one uses R5 anymore”, etc. I have and still do work in IT with server arrays, etc. That is not the point though – unless you work with –every– single company, no-one here is qualified to say what “everyone” or ‘most’ companies/engineers are doing “these days” and it makes 0 sense to claim it.

  12. @andrew Actually I think you’ll find that you have the premise backwards. My entire point was that since R0 requires one fewer drives, it is less likely (for the same capacity and performance) to experience a drive failure, at all. So rather than being focused on what happens after a failure like you thought, the entire point I was making here was the opposite, that you have to stop dealing only with that one portion of the risk and look at the risk as a whole. You can’t overlook the trigger to the rebuild and only look at the rebuild, you have to look at the full risk scenario. Just run the math, it’s that simple.

    And yes, I thought that I did show exactly what you are stating that I should show… that by having R5 you clearly increase the risk of the initial failure, by a lot. That’s the fundamental premise here. I’m not sure how you missed that, as that was the entire basis of the article.

    R5 requires N+1 disks compared to the N disks of RAID 0. That gives equal capacity and a rather significant performance advantage to R0 under most performance scenarios. So the R0 has one fewer disk to fail. That is generally pretty statistically significant in any real world sized array.

    That’s before we take into account, and allows us to ignore, the accepted industry knowledge that R5 increases wear and tear causing each individual drive to be more likely to experience failure as well. This is caused slightly from increased wear and tear during reads, but almost entirely by the effects of the 4x write expansion. The drives are read more often and written dramatically more often than its R0 counterpart. Often 200 – 400% more. That’s a big number for wear and tear.

    So take the incredibly obvious increased initial risk of having more moving parts to fail. Then work them drastically harder under the same workload and you’ve got a super clear picture of why the R5 array, apples to apples, suffers initial drive failure more often than the R0 array.

    Now that we’ve established why R5s need to recover more often, then when we talk about how reliably R5 rebuild operations are, we have a clear understanding that this must be significantly reliable enough to overcome the additional risk posed by the increased initial failures that they cause.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.