Doing IT at Home

Doing IT at Home: Enterprise Networking

6 Comments

In my fifth installment in the continuing series on Doing IT at Home I would like to focus on enterprise networking.  Many ways in which we can bring business class IT into our homes can really be done for free but networking is sadly, not one of those areas but it does not have to be as costly as you may, at first, think and having a good, solid, enterprise-class home network can bring many features that other IT at Home projects do not.

Implementing an real, working business class network at home lays the foundation for a lot of potential learning, experimenting, testing and growth; and compared to other, small and less ambitious projects this one will likely shine very brightly on a curriculum vitae.

Now we have to start by defining what we mean by “enterprise networking.”  Clearly the needs and opportunity for networking at home are not the same as they are in a real business, especially not a large one – at least without resorting to a pure lab setup which is not our goal of bringing IT home.  Having a lab at home is excellent and I highly recommend it, but I would not recommend building to “true lab” in your home until you have truly taken advantage of the far better opportunity to treat your home as a production “living lab” environment.  Needing your “living lab” to be up and running, in use every day changes how you view it, how you treat it and what you will take away from the experience.  A pure lab can be very abstract and it is easy to treat it in such a way that much of the educational opportunity is lost.

There are many aspects of enterprise networking that make sense to apply to our homes.  Every home is different and I will only present some ideas and I would love to hear what others can come up with as interesting ways to take home networking to the next level.

Firewall or Unified Thread Management (UTM):  This is the obvious starting point for upgrading any home network.  Most homes use a free multi-purpose device that is provided by their ISP that lacks features and security.  The firewall is the most featureful networking device that you will use in a home or in a small business and is the most important for providing basic security.  Your firewall provides the foundation of your home or small business network so getting this in place first makes sense.

There are numerous firewall and UTM products on the market.  Even for home or SMB use you will be flush with options.  You can only practically use a single unit and you need one powerful enough to be able to handle the throughput of a consumer WAN connection which may be a challenge with some vendors as consumer Internet access is getting very fast and requires quite a bit of processing power, especially from a UTM solution.

Choosing a firewall will likely mostly come down to your career goals and and price.  If you hope to pursue a career or certification in Cisco, Juniper or Palo Alto, for example, you will want to get devices from those vendors that allow you to do training at home.  These will be very expensive options but if that career path is your chosen one, having that gear at home will be immensely valuable not only for your learning and testing but for interviewing as well.

If you don’t have specific security or networking career goals your options are more open. There are traditional small business firewall suppliers like Netgear ProSafe that are low cost and easy to manage.  UTM devices are starting to enter this market like Netgear ProSecure, but these are almost universally more costly.  There is the software-only approach where you provide your own hardware and build the firewall yourself.  This is very popular and has many good options for software including pfSense, SmoothWall, Untangle and VyOS.  These vary in features and complexity.  For most cases, however, I would recommend a Ubiquiti EdgeMax router which runs Brocade Vyatta firmware.  These are less costly than most UTMs and run enterprise routing and firewall firmware – outside of needing a specific vendor’s product for networking career goals, this is the best learning, security and feature value on the market and will allow learning nearly any firewall or router skills outside of those specific to proprietary vendors.

When starting down the path of enterprise networking at home, remember to consider if you should also begin acquiring rackmount gear rather than tabletop equipment.  Having a racks, possibly just a half or even a quarter rack or cabinet, at home can make doing home enterprise projects much easier and can make the setup much more attractive, in many cases and is, like many of these projects, just that much more impressive.  Consider this when buying gear because firewalls in this category, are the hardest to find in rackmount configurations, much to my chagrin.

Switch: It has become common in home networking to begin forgoing a physical switch in favor of pure wireless solutions and for many homes where networking is not core to function this may make perfect sense.  But for us, it likely does not.  Adding switching makes for more learning opportunities, a better showcase, far more flexibility, faster data transfers inside of the house, a great number of connections and better reliability.  For a normal home user with few devices, most of which are mobile ones, this would be a waste, but for an IT pro at home, a real switch is practically a necessity.

Switching come in three key varieties.  Unmanaged, or dumb switches, which is all that you would find in a home or most small businesses.  Basic connectivity but nothing more.  This might be all that you need if you do not intend to explore deeper learning opportunities in networking.

Smart switching is a step up from an unmanaged switch.  A smart switch is often very low cost but adds additional features, normally through a web interface, that allow you to actively manage the switch, change configurations, troubleshoot, great VLANs and QoS, monitor, etc.  For someone looking to step up their at home network and approach networking from a higher-end small business perspective this is a great option and a very practical one for a home.

Managed switches are the most enterprise and by far the most costly.  These use SNMP and other standard protocols for remote management and monitoring and generally have the most features although often Smart switches have just as many.  Managed switches are not practical in a home for any reason as their benefits are around scalability, not features, but, like with everything, if learning those features is a key goal then this is another place where spending more money not only for those features but also to get “name brand” switches, like Cisco, Juniper, Brocade or HP can be an important investment.  But if the goal is only to learn the tools and standards of managed switches and not to go down the path of learning a specific implementation then lower cost options like Netgear Prosafe might make sense.

Once we decide on unmanaged, smart or managed switches then we have to decide on the “layer” of the switch.  This also has three options: Layer 2, Layer 2+ and Layer 3.  For home and small business use, L2 switches are the most common.  I have never seen more than an L2 in a home and rarely in a small business.  L2 are traditional switches that handle only Ethernet switching.  You can create VLANs on L2 switches but you cannot route traffic between the VLANs, that would require a router.  An L2+ switch adds some inter-VLAN traffic handling to allow VLANs to exist using static routes.  L3 switches have full IP handling and can do dynamic routing protocols.

So if you need to study “large” scale routing, an L3 switch is good.  This is not a common need and would be the most expensive route and would imply that you intend to purchase a lot more networking gear than just one switch.  In a home lab, this might exist, for handling the home itself, it would not.  If you want to implement VLANs in your home, perhaps one LAN, one Voice LAN, a DMZ and one Guest LAN then an L2+ switch is ideal.  If you don’t plan to study VLANing, stick to L2.

Cabling: One aspect of home networking that is far too often overlooking is implementing a quality cabling plant inside the home.  This requires far more effort than other home networking projects and falls more into the electrician space rather than the IT professional space but is also one of the most important pieces from the home owner perspective and end user perspective rather than the IT pro perspective.  A good, well installed cabling plant will make a home more attractive to buyers and make the value a powerful home network even better.

If you live in an apartment, likely you do not have the option to alter the wiring in this way, unfortunately.  But for home owners, cabling the house can be a great project with a lot of long term value.  Setting up a well labeled and organized cabling plant, just like you would in a business, can be attractive, impressive and eminently useful.  With good, well labeled cabling you can provide high speed, low latency connections without the need for wireless to every corner of your home.  I have found that cabling bedrooms, entertainment spaces and even the kitchen are very valuable.  This allows for higher throughput communications to all devices as wireless congestion is relieved and wired throughput is preferred when possible.  Devices such as video game consoles, smart televisions, receivers, media appliances (a la AppleTV, Roku, Google), desktops, docking stations, stationary laptops, VoIP phones and more all can benefit from the addition of complete cabling.

Wireless Access Point: These days home networks are primarily wireless with many homes being exclusively wireless.  Even if you follow my advice and have great wired networking you still need wireless whether for smart phones, tablets, laptops, guest access or whatever.    A typical home network will already have some cheap, probably unreliable wireless from the onset.  But I propose at least a moderate upgrade to this as a good practice in home networking.

Enterprise Access Points have come down in price dramatically today and a few vendors have even gotten then below one hundred dollars for high quality, centrally managed devices.  Good devices have high quality radios and antennae that will improve range and reliability.  Generally they will come with extra features like mapping, monitoring, centralized management console, VLAN support, hotspot login options, multiple SSID support, etc.  Most of these features are not needed in a home network but are commonly used even in a small business and having them at home for such a low price point makes sense.  If you own a large home, using good Access Points with centralized management can be additionally beneficial in providing whole home coverage.

Having secure guest access via the access point can be very nice in a home allowing guests to be isolated from the data and activities on the home network.  No need to share private passwords and provide access to data that is not necessary while still allowing guests to connect their mobile phones and tablets.  An ever more important feature.

If your home includes outdoor space, adding wireless projects to provide outdoor coverage could also make fora  great learning project.  Outdoor access points and specialized antenna can make for a fun and very useful project.  Make yourself able to stay connected even while roaming outdoors.

Good, enterprise access points are often quite attractive as well, being designed to be wall or ceiling mounted, making it easier to put them in good placement locations to better cover your available space.

Power over Ethernet: Now that you are looking at deploying enterprise access points and if you followed by earlier article on doing a PBX at home and you have desktop or wall mount VoIP phones you may want to consider adding additional PoE switching to reduce the need for electrical cables or power injectors.  A small PoE switch is not expensive and, while never really necessary, can make your home network that much more interesting and “polished.”  Many security devices take advantage of PoE as well as do some project board computers that are increasingly popular today.  The value to adding PoE is ever increasing.

Network Software: Once your home is upgraded to this level, it is only natural to then bring in network management and monitoring software to leverage it even further.  This could be as simple as setting up Wireshark to look at your LAN traffic or it could mean SNMP Monitors, Netflow tools and the like. What is available to you is highly dependent on the vendors and products that you choose but the options are there and this is really where much of the benefit comes in regards to the ongoing educational aspects of network.  Building the network and performing the occasional maintenance will, of course, be very good experience but having the tools to now watch the living network at work and learn from it will be key to the continuing value beyond the impressive end user experience that your household will enjoy.

Storage

The Cult of ZFS

9 Comments

It’s pretty common for IT circles to develop a certain cult-like or “fanboy” mentality.  What causes this reaction to technologies and products I am not quite sure, but that it happens is undeniable.  One area that I never thought that I would see this occur is in the area of filesystems – one of the most “under the hood” system components and one that, until recently, received literally no attention even in decently technical circles.  Let’s face it, misunderstanding when something comes from Active Directory versus from NTFS is nearly ubiquitous.  Filesystems are, quite simply, ignored.  Ever since Windows NT 4 released and NTFS was the only viable option the idea that a filesystem is not an intrinsic component of an operating system and that there might be other options for file storage has all but faded away.  That is, until recently.

The one community where, to some small degree, this did not happen was the Linux community, but even there Ext2 and its descendants so completely won mindshare that even thought they were widely available, alternative filesystems were sidelines and only XFS received any attention, historically, and even it received very little.

Where some truly strange behavior has occurred, more recently, is around Oracle’s ZFS filesystem, originally developed for the Solaris operating system and the X4500 “Thumper” open storage platform (originally under the auspices of Sun prior to the Oracle acquisition.)  At the time (nine years ago) when ZFS released, competing filesystems were mostly ill prepared to handle large disk arrays that were expected to be made over the coming years.  ZFS was designed to handle them and heralded in the age of large scale filesystems.  Like most filesystems at that time, ZFS was limited only to a single operating system and so, while widely regarded as a great leap forward in filesystem design, it produces few ripples in the storage world and even fewer in the “systems” world where even Solaris administrators generally considered it a point of interest only for quite some time, mostly choosing to stick to the tried and true UFS that they had been using for many years.

ZFS was, truly, a groundbreaking filesystem and I was, and remain, a great proponent of it.  But it is very important to understand why ZFS did what it did, what its goals are, why those goals were important and how it applies to us today.  The complexity of ZFS has lead to much confusion and misunderstanding about how the filesystem works and when it is appropriate to use.

The principle goals of ZFS were to make a filesystem capable of scaling well to very large disk arrays.  At the time of its introduction, the scale to which ZFS was capable was unheard of in other file systems but there was no real world need for a filesystem to be able to grow that large.  By the time that the need arose, many other file systems such as NTFS, XFS, Ext3 and others had scaled to accommodate the need.  ZFS certainly lead the charge to larger filesystem handling but was joined by many others soon thereafter.

Because ZFS originated in the Solaris world where, like all big iron UNIX systems, there is no hardware RAID, software RAID had to be used.  Solaris had always had software RAID available as its own subsystem.  The decision was made to build a new software RAID implementation directly into ZFS.  This would allow for simplified management via a single tool set for both the RAID layer and the filesystem.  It did not introduce any significant change or advantage to ZFS, as is often believed, it simply shifted the interface for the software RAID layer from being its own command set to being part of the ZFS command set.

ZFS’ implementation of RAID introduced variable width stripes in parity RAID levels.  This innovation closed a minor parity RAID risk known as the “write hole”.  This innovation was very nice but came very late as the era of reliable parity RAID was beginning to end and the write hole problem was already considered to be an unmentioned “background noise” risk of parity arrays as it was not generally considered a thread due to its elimination through the use of battery backed array caches and, at about the same time, non-volatile array caches – avoid power loss and you avoid the write hole.  ZFS needed to address this issue because, as software RAID, it was at greater risk to the write hole than hardware RAID is because there is no opportunity for a cache protected against power loss – hardware RAID offers the potential for an additional layer of power protection for arrays.

The real “innovation” that ZFS inadvertently made was that instead of just implementing the usual RAID levels of 1, 5, 6 and 10 they instead “branded” these levels with their own naming conventions.  RAID 5 is known as RAIDZ.  RAID 6 is known as RAIDZ2.  RAID 1 is just known as mirroring.  And so on.  This was widely considered silly at the time and pointlessly confusing but, as it turned out, that confusion because the cornerstone of ZFS’ revival many years later.

It needs to be noted that ZFS later added the industry’s first production implementation of a RAID 7 (aka RAID 7.3) triple parity RAID system and branded it RAIDZ3.  This later addition is an important innovation for large scale arrays that need the utmost in capacity while remaining extremely safe but are willing to sacrifice performance in order to do so.  This remains a unique feature of ZFS but one that is rarely used.

In the spirit of collapsing the storage stack and using a single command set to manage all aspects of storage the logical volume management functions were rolled into ZFS as well.  It is often mistakenly believed that ZFS introduced logical volume management in certain circles but nearly all enterprise platforms, including AIX, Linux, Windows and even Solaris itself, had already had logical volume management for many years.  ZFS was not doing this to introduce a new paradigm but simply to consolidate management and wrap all three key storage layers (RAID, logical volume management and filesystem) into a single entity that would be easier to manage and could provide inherent communications up and down the stack.  There are pros and cons to this method and an industry opinion remains unformed nearly a decade later.

One of the most important aspects of this consolidation of three systems into one is that now we have a very confusing product to discuss.  ZFS is a filesystem, yes, but it is not only a filesystem.  It is a logical volume manager, but not only a logical volume manager.  People refer to ZFS as a filesystem, which is its primary function, but that it is so much more than a filesystem can be very confusing and makes comparisons against other storage systems difficult.  At the time I believe that this confusion was not foreseen.

What has resulted from this confusing merger is that ZFS is often compared to other filesystems, such as XFS or Ext4.  But this is confusing as ZFS is a complete stack and XFS is only one aspect of a stack.  ZFS would be better compared to MD (Linux Software RAID) / LVM / XFS or to SmartArray (HP Hardware RAID) / LVM/ XFS than to XFS alone.  Otherwise it appears that ZFS is full of features that XFS lacks but, in reality, it is only a semantic victory.  Most of the features often touted by ZFS advocates did not originate with ZFS and were commonly available with the alternative filesystems long before ZFS existed. But it is hard to compare “does your filesystem do that” because the answer is “no…. my RAID or my logical volume manager do that.”  And truly, it is not ZFS the filesystem providing RAIDZ, it is ZFS the software RAID subsystem that is doing so.

In order to gracefully handle very large filesystems, data integrity features were built into ZFS which included a checksum or hash check throughout the filesystem that could leverage the inclusive software RAID to repair corrupted files.  This was seen as necessary due to the anticipated size of ZFS filesystems in the future.  Filesystem corruption is a rarely seen phenomenon but as filesystems grow in size the risk increases.  This lesser known feature of ZFS is possibly its greatest.

ZFS also changed how filesystem checks are handled.  Because of the assumption that ZFS will be used on very large filesystems there was a genuine fear that a filesystem check on boot time could take impossibly long to complete and so an alternative strategy was found.  Instead of waiting to do a check at reboot the system would require a scrubbing process to run and perform a similar check while the system was running.  This requires more system overhead while the system is live but the system is able to recover from an unexpected restart more rapidly.  A trade off but one that is widely seen as very positive.

ZFS has powerful snapshotting capabilities in its logical volume layer and in its RAID layer has implemented very robust caching mechanisms making ZFS an excellent choice for many use cases.  These features are not unique in ZFS but are widely available in systems older than ZFS.  They are, however, very good implementations of each and very well integrated due to ZFS’ nature.

At one time, ZFS was open source and during that era its code became a part of Apple’s Mac OSX and FreeBSD operating systesms because they were compatible with the ZFS license.  Linux did not get ZFS at that time due to challenges around licensing.  Had ZFS licensing allowed Linux to use it unencumbered the Linux landscape would likely be very different today.  Mac OSX eventually dropped ZFS as it was not seen as having enough advantages to justify it in that environment.  FreeBSD clung to ZFS and, over time, it became the most popular filesystem on the platform although UFS is still heavily used as well.  Oracle closed the source of ZFS after the Sun acquisition leaving FreeBSD without continuing updates to its version of ZFS while Oracle continued to develop ZFS internally for Solaris.

Today Solaris remains using the original ZFS implementation now with several updates since its split with the open source community.  FreeBSD  and others continued using ZFS in the state it was when the code was closed source, no longer having access to Oracle’s latest updates.  Eventually work to update the abandoned open source ZFS codebase was taken up and is now known as OpenZFS.  OpenZFS is still fledgling and has not yet really made its mark but has some potential to revitalize the ZFS platform in the open source space but at this time, OpenZFS still lags ZFS.

Open source development for the last several years in this space has focused more on ZFS’ new rival BtrFS which is being developed natively on Linux and is well supported from many major operating system vendors.  BtrFS is very nascent but is making major strides to reach feature parity with ZFS in implemented features but has large aspirations and due to ZFS’ closed source nature has the benefit of market moment.  BtrFS was started, like ZFS, by Oracle and has been widely seen as Oracle’s view of the future being a replacement for ZFS even at Oracle.  At this time BtrFS has already, like ZFS, merged the filesystem, logical volume management and software RAID layers, implemented checksumming for filesystem integrity, scales even larger than ZFS (same absolute limit but handles more files), copy on write snapshots, etc.

ZFS, without a doubt, was an amazing filesystem in its heyday and remains a leader today.  I was a proponent of it in 2005 and I still believe heavily in it.  But it has saddened me to see the community around ZFS take on a fervor and zealotry that  does it no service and makes the mention of ZFS almost seem as a negative – ZFS being so universally chosen for the wrong reasons: primarily a belief that its features exist nowhere else, that its RAID is not subject to the risks and limitations that those RAID levels are always subject to or that it was designed for a different purpose (primarily performance) other than what it was designed for.  And when ZFS is a good choice, it is often implemented poorly based on untrue assumptions.

ZFS, of course, is not to blame.  Nor, as far as I can tell, are its corporate supporters or its open source developers.  Where ZFS seems to have gone awry is in a loose, unofficial community that has only recently come to know ZFS, often believing it to be new or “next generation” because they have only recently discovered it.  From what I have seen this is almost never via Solaris or FreeBSD channels but almost exclusively smaller businesses looking to use a packaged “NAS OS” like FreeNAS or NAS4Free who are not familiar with UNIX OSes.  The use of packaged NAS OSes, primarily by IT shops that possess neither deep UNIX nor storage skills and, consequently, little exposure to the broader world of filesystems outside of Windows and often little to no exposure to logical volume management and RAID, especially software RAID at all, appears to lead to a “myth” culture around ZFS with it taking on an almost unquestionable, infallible status.

This cult-like following and general misunderstanding of ZFS leads often to misapplications of ZFS or a chain of decision making based off of bad assumptions that can lead one very much astray.

One of the most amazing changes in this space is the change in following from hardware RAID to software RAID.  Traditionally, software RAID was a pariah in Windows administration circles without good cause – Windows administrators and small businesses, often unfamiliar with larger UNIX servers believed that hardware RAID was ubiquitous when, in fact, larger scale systems always used software RAID.  Hardware RAID was, almost industry wide, considered a necessity and software RAID completely eschewed.  That same audience, now faced with the “Cult of ZFS” movement, now react in exactly the opposite way believing that hardware RAID is bad and that ZFS’ software RAID is the only viable option.  The shift is dramatic and neither approach is valid – both hardware and software RAID and both in many implementations are very valid options and even using ZFS the use of hardware RAID might easily be appropriate.

ZFS is often chosen because it is believed that it is the highest performance option in filesystems but this was never a key design goal of ZFS.  The features allowing it to scale so large and handle so many different aspects of storage actually make being highly performant very difficult.  ZFS, at the time of its creation, was not even expected to be as fast as the venerable UFS which ran on the same systems as it.  However, this is often secondary to the fact that filesystem performance is widely moot as all modern filesystems are extremely fast and filesystem speed is rarely an important factor – especially outside of massive, high end storage systems on a very large scale.

An interesting study of ten filesystems on Linux produced by Phoronix in 2013 showed massive differences in filesystems by workload but no clear winners as far as overall performance.  What the study showed conclusively is that matching workload to filesystem is the most important choice, that ZFS falls to the slower side of all mainstream filesystems even in its more modern implementations and that choosing a filesystem for performance reasons without a very deep understanding of the workload will result in unpredictable performance – no filesystem should be chosen blindly if performance is an important factor.  Sadly, because the test was done on Linux, it lacked UFS which is often ZFS’ key competitor especially on Solaris and FreeBSD and it lacked HFS+ from Mac OSX. 

Moving from hardware RAID to software RAID carries additional, often unforeseen risks to shops not experienced in UNIX as well.  While ZFS allows for hot swapping, it is often forgotten that hot swap is primarily a feature of hardware, not of software, and it is also widely unknown that blind swapping (removal of hard drives without first offlining them in the operating system) is not synonymous with hot swapping and this can lead to disasters for shops moving from a tradition of hardware RAID that handled compatibility, hot swap and blind swapping transparently for them to a software RAID system that requires much more planning, coordination and understanding of the system in order to use safely.

A lesser, but still common misconception of ZFS, is that it is a clustered filesystem suitable for use on shared DAS or SAN scenarios a la OCFS, VxFS and GFS2.  ZFS is not a clustered filesystem and shares the same limitations in that space as all of its common competitors.

ZFS can be an excellent choice but it is far from the only one.  ZFS comes with large caveats, not the least of which is the operating system limitations associated with it, and while it has many benefits few, if any, are unique to ZFS and it is very rare that any shop will benefit from every one of them.  As with any technology, there are trade offs to be made.  One size does not fit all.  The key to knowing when ZFS is right for you is to understand what ZFS is, what is and is not unique about it, what its design goals are, how comparing a storage storage to a pure filesystem produces misleading results and what inherent limitations are tied to it.

ZFS is a key consideration and the common choice when Solaris or FreeBSD is the chosen operating system.  With rare exception, the operating system should never be chosen for ZFS, but instead ZFS should be often chosen, but not always, when the operating system is chosen.  The OS should drive the filesystem choices in all but the rarest of cases.  The choice of operating system is so dramatically more important than the choice of filesystem.

ZFS can be used on Linux but is not considered an enterprise option there but more of a hobby system for experimentation as no enterprise vendor (such as Red Hat, Suse or Canonical) support ZFS on Linux and as Linux has great alternatives already.  Someday ZFS might be promoted to a first class filesystem in Linux but this is not expected as BtrFS has already entered the mainline kernel and been included in production releases by several major vendors.

While ZFS will be seen in the vast majority of Solaris and FreeBSD deployments, this is primarily because it has moved into the position of default filesystem and not because it is clearly the superior choice in those instances or has even been evaluated critically.  ZFS is perfectly well suited to being a general purpose filesystem where it is native and supported.

What is ZFS’ primary use case?

ZFS’ design goal and principal use case is for Solaris and FreeBSD open storage systems providing either shared storage to other servers or as massive data repositories for locally installed applications.  In these cases, ZFS’ focus on scalability and data integrity really shine.  ZFS leans heavily towards large and enterprise scale shops and generally away from applicability in the small and medium business space where Solaris and FreeBSD skills, as well as large scale storage needs, are rare.

Reference: http://www.phoronix.com/scan.php?page=article&item=linux_310_10fs&num=1

 

Storage

Understanding the Western Digital SATA Drive Lineup (2014)

4 Comments

I choose to categorize Western Digital’s SATA drive lineup for several reasons. One is that WD is the current market leader in spinning hard drives so this makes the categorization most useful to the greatest number of people, the “color coded” line is, based on anecdotal evidence, far and away the chosen drive family of the small business market where the diagnosis is most important and SATA drives retain the most disparity of features and factors making them far more necessary to understand well. While technically the only difference between a SAS (SCSI) and SATA (ATA) drive or even a Fibre Channel (FC) drive is nothing but the communications protocol used to communicate with them, in practical terms SAS and FC drives are only made in certain, high reliability configurations and do not require the same degree of scrutiny and do not carry the same extreme risks as SATA drives. Understanding SATA drive offerings is the more important for practical, real world storage needs.

WD has made understanding their SATA drive line up especially easy by adding color codes to the majority of their SATA drive offerings – those deemed to be “consumer” drives, and an “E” designation on their enterprise SATA drives and one outlier, the high performance Velociraptor drives which seek to compete with common SAS performance for SATA controllers. Altogether they have seven SATA drive families to consider covering the gamut of drive factors. While this diagnosis will apply to the easy to understand WD lineup, by comparing factors here with the offerings of other drive makers the use cases of their drives can be determined as well.

In considering SATA drives, three really key factors stand out as being the most crucial to consider (outside of price, of course.)

URE Rate: URE, or Unrecoverable Read Error, is an event that happens, with some regularity, to electromechanical disk storage media where a single sector is unable to be retrieved. In a standalone drive this happens from time to time but generally only affects a single file and users typically see this as a lost file (often one they do not notice) or possible a corrupt filesystem which may or may not easily be corrected. In healthy RAID arrays (other than RAID 0), the RAID system provides mirroring and/or parity that can cover for this sector failure and recreate the data protecting us from URE issues. When a RAID array is in a degraded state UREs are a potential risk again. In its worst case, a URE on a degraded parity array can, in some cases, cause total loss of an array (all data is lost.) So considering UREs and their implications in any drive purchases is extremely important and is the primary driver of cost differential in drives of varying types. URE varies from the low end at 10^14 to the high end at 10^16. The numbers are so large that they are always written in scientific notation. I will not go into an in-depth explanation of URE rates, ramifications and mitigation strategies here, but understanding URE is critical to decision making around drive purchases, especially in the large capacity, lower reliability space of SATA drives.

Spindle Speed: This is one of the biggest factors to most users, spindle speed directly correlates to IOPS and throughput. While measurements of drive speed are dynamic, at best, spindle speed is the best overall way to compare two otherwise identical drives under identical load. A 15,000 RPM drive will deliver almost exactly double the IOPS and throughput of a 7,200 RPM drive, for example. SATA drives commonly come in 5,400 RPM and 7,200 RPM varieties with rare high performance drives available at 10,000 RPMs.

Error Recovery Control (ERC): Also known as TLER (Time Limited Error Recovery) in WD parlance, ERC is a feature of a drive’s firmware which allows for configurable time limits for read or write errors which can be important when a hard drive is used in a RAID array as often error recovery needs to be handled at the array, rather than the drive, level. Without ERC, a drive is more likely to be incorrectly marked as failed when it has not. This is most dangerous in hardware based parity RAID arrays and has differing levels of effectiveness based on individual RAID controller parameters. It is an important feature for drives assumed for use in RAID arrays.

In addition to these key factors, WD lists many others for their drives such as cache size, number of processors, mean time between failures, etc. These tend to be far less important, especially MTBF and other reliability numbers as these can be skewed or misinterpreted easily and rarely offer the insight into drive reliability that we expect or hope. Cache size is not very significant for RAID arrays as they need to be disabled for reasons of data integrity. So outside of desktop use scenarios, the size of a hard drive’s cache is generally considered irrelevant. CPU count could also be misleading as single CPUs may be more powerful than dual CPUs if the CPUs are not identical and the efficacy of the second CPU is unknown. But WD lists this as a prominent feature of some drives and it is assumed that there is measurable performance gain, most likely in latency reduction, through the addition of the second CPU. I do, however, continue to treat this as a trivial factor and mostly only useful as a point of interest rather than as a decision factor
The drives.

All color-coded drives (Blue, Green, Red and Black) share one common factor – they have the “consumer” URE rating of 10^14. Consumer is a poor description here but is, more or less, industry standard. A better description is “desktop class” or suitable for non-parity RAID uses. The only truly poor application of 10^14 URE drives is in parity RAID arrays and even there, they can have their place if properly understood.

Blue: WD Blue drives are the effective baseline model for the SATA lineup. They spin at the “default” 7,200 RPMs, lack ERC/TLER and have a single processor. Drive cache varies between 16MB, 32MB and 64MB depending on the specific model. Blue drives are targeted at traditional desktop usage – as single drives with moderate speed characteristics, not well suited to server or RAID usage. Blue drives are what is “expected” to be found in off the shelf desktops. Blue drives have widely lost popularity and are often not available in larger sizes. Black and Green drives have mostly replaced the use of Blue drives, at least in larger capacity scenarios.

Black: WD Black drives are a small upgrade to the Blue drives changing nothing except to upgrade from one to two processors to slightly improve performance while not being quite as cost effective. Like the Blue drives they lack ERC/TLER and spin at 7,200 RPM. All Black drives have the 64MB cache. As with the Blue drives, Black drives are most suitable for traditional desktop applications where drives are stand alone.

Green: WD Green drives, as their name nominally implies, are designed for low power consumption applications. They are most similar to Blue drives but spin at a slower 5,400 RPMs which requires less power and generates less heat. Green drives, like Blue and Black, are designed for standalone use primarily in desktops that need less drive performance than is expected in an average desktop. Green drives have proven to be very popular due to their low cost of acquisition and operation. It is assumed, as well, that Green drives are more reliable than their faster spinning counterparts due to the lower wear and tear of the slower spindles although I am not aware of any study to this effect.

Red: WD Red drives are unique in the “color coded” WD drive line up in that they offer ERC/TLER and are designed for use in small “home use” server RAID arrays and storage devices (such as NAS and SAN.) Under the hood the WD Red drives are WD Green drives, all specifications are the same including the 5,400 RPM spindle speed, but with TLER enabled in the firmware. Physically they are the same drives. WD officially recommends Red drives only for consumer applications but Red drives, due to their lower power consumption and TLER, have proven to be extremely popular in large RAID arrays, especially when used for archiving. Red drives, having URE 10^14, are dangerous to use in parity RAID arrays but are excellent for mirrored RAID arrays and truly shine at archival and similar storage needs where large capacity and low operational costs are key and storage performance is not very important.

Outside of the color coded drives, WD has three SATA drive families which are all considered enterprise. What these drives share in common is that their URE rate is much higher than that of the “consumer” color coded drives. Ranging from URE 10^15 to 10^16 depending on model. The most important result of this URE rate is that these drives are far more applicable to use in parity RAID arrays (e.g. RAID 6.)

SE: SE drives are WD’s entry level enterprise SATA drives with URE 10^15 rates and 7,200 RPM spindle speeds. They have dual processors and a 64MB cache. Most importantly, SE drives have ERC/TLER enabled. SE drives are ideal for enterprise RAID arrays both mirrored and parity.

RE: RE drives are WD’s high end standard enterprise SATA drives with all specifications being identical to the SE drives but with the even better URE 10^16 rate. RE drives are the star players in WD’s RAID drive strategy being perfect for extremely large capacity arrays even when used in parity arrays. RE drives are available in both SATA and SAS configurations but with the same drive mechanics.

Velociraptor: WD’s Velociraptor is a bit of an odd member of the SATA category. With URE 10^16 and a 10,000 RPM spindle speed the Velociraptor is both highly reliable and very fast for a SATA drive competing with common, mainline SAS drives. Surprisingly, the Velociraptor has only a single processor and even more surprisingly, it lacks ERC/TLER making it questionable for use in RAID arrays. Lacking ERC, use in RAID can be considered on an implementation by implementation basis depending on how the RAID system interacts with the drive’s timing. With the excellent URE rating, Velociraptor would be an excellent choice for large, higher performance parity RAID arrays but only if the array handles the error timing in a graceful way, otherwise the risk of the array marking the drive as having failed is unacceptably high for an array as costly as this would be. It should be noted that Velociraptor drives do not come in capacities comparable to the other SATA drive offerings – they are much smaller.

Of course the final comparison that one needs to make is in price. When considering drive purchases, especially where large RAID arrays are concerned or for other bulk storage needs, the per drive cost is often a major, if not the driving, factor. The use of slower, less reliable drives in a more reliable RAID level (such as Red drives in RAID 10) versus faster, more reliable drives in a less reliable RAID level (such as RE drives in RAID 6) often provides a better blend of reliability, performance, capacity and cost. Real world drive prices play a significant factor in these decisions. These prices, unlike the drive specifications, can fluctuate from day to day and swing planning decisions in different directions but, overall, tend to remain relatively stable in comparison to one another.

At the time of this article, at the end of 2013, a quick survey of prices of 3TB drives from WD give these approximate breakdown:

Green $120
Red $135
Black $155
SE $204
RE $265

As can be seen, the jump in price primarily comes between the consumer or desktop class drives and the enterprise drives with their better URE rates with Red and RE drives, both with ERC/TLER, being in a price ratio of almost exactly 2:1 making, for equal capacity, it favorable to choose many more Red drives in RAID 10 than fewer RE drives in RAID 6, as an example. So comparing a number of factors, along with current real world prices, is crucial to making many buying decisions.

Newer drives, just being released, are starting to see reductions in onboard drive cache for exactly the reasons we stated above, drives designed around RAID use have little or no purpose to having onboard cache as it needs to be disabled for data integrity purposes.

Drive makers today are offering a wide variety of traditional spindle-based drive options to fit many different needs. Understanding these can lead to better reliability and more cost effective purchasing and will extend the usefulness of traditional drive technologies into the coming years.

Best Practices

Choosing Software Versions for Deployment

Leave a comment

Something that I see discussed very often in IT circles is “which version of software should I install.” This could apply to a database, an application, firmware or, probably most often, operating systems, and with the upcoming end of support life for Windows XP the topic has reached a fevered pitch.

There are effectively two sides to this discussion. One side believes that the latest and, presumably, greatest software should always be used. The other believes that software needs to mature and take a “wait and see” approach or even considers each version to be a different product and not a continuum of development.

Both approaches have their merits and neither should exist completely without the other. Blindly updating software willy nilly is not wise and avoiding patches and updates without reason is not wise either. Careful consideration of the factors and empathy for the software development process are important to keep in mind when making these decisions.

First, there are two completely different scenarios to consider. One is the updating of current, existing software. The assumption being that the current state of things is “working” with the accepted possibility that “working” might include a security exposure that has been discovered and requires updating in order to close. The other scenario is a new deployment where there is nothing currently and, we are starting from scratch.

Let’s start with the second case, as it is far easier to provide guidance on.

In the case of new software deployments (or new operating systems), always use the current, most recent version of the software unless there is a clearly known technology limitation preventing it ?? such as known bugs or software incompatibilities.

Software is not like other types of products, especially not in today’s world of online patch releases and updates. I assume that the mentality that old versions of software might be preferable to current ones comes from a combination of physical products (watches, cars, dishes, furniture, wine) where a specific year or model might be superior to a newer model for various reasons and from legacy software delivery modes where finished software products were just ??thrown over the wall? and the final state was, quite simply, the final state without any reasonable opportunities for updates, patches or fixes. Neither of these cases applies to modern business software (with only the rarest of exceptions.)

Software development is roughly a continuum. Normal development processes have new software being built on top of old software either directly (by creating updates to an existing code base) or indirectly (by rebuilding based on knowledge gained from having built a previous version of the software.) The idea being that each subsequent version of software is superior to the one preceding it. This is not guaranteed, of course, there are such concepts as regression errors and just bad development, but by and large, software improves over time – especially when we are talking about enterprise class software used in businesses and under active development. New software is not just the next phase of the old software, it also represents, in nearly all cases, the current state of patches, bug fixes, updates and, when necessary, changes in approach or technique. New software, coming from quality shops, is almost exclusively better than old software. Software evolves and matures.

Beyond the quality of software itself, there is the concept of investing in the future. Software is not something that can sit on the shelf forever. It needs to stay, to some degree, up to date or it stops functioning because the platform that it runs on changes, some new artifact comes to light, security holes are discovered or needs change. Installing old software means that there is an investment in the past, an investment in installing, learning, using and supporting old technology. This is called “technical debt.” This old technology might last for years or even decades, but old software loses value over time and becomes increasingly expensive to support both for the vendors, if they continue to support it, and for the end users, who have to support it.

The same concept of technical debt applies to the software vendors in question. There is a very large cost in creating software and especially in maintaining multiple versions of that software. Software vendors have a lot of incentive to reduce support for older versions to focus resources on current software releases (this is a major reason why SaaS deployments are so popular, the vendor controls the available versions and can eliminate legacy versions through updates.) If customers require support for old versions, the cost must be absorbed somewhere and often it is absorbed both in monetary impact to all customers as well as a decrease in focus on the new product as development teams must be split to support patching old versions as well as developing the new. The more effort that must go in to old versions, the less effort that can be put into new improvements.

Within the framework of what I have already said, it is important to talk about code maturity. Often code maturity is stated as a reason for deploying “old code”, but I think that this is an IT misunderstanding of software development processes. If we think about a released line of code, just because it is released and in use does not really make it more mature. Code does not change in the wild, it just sits there. Its maturity is “locked” on the day that it is released. If it is patched, then yes, it would “mature” post release. Later versions of the same software, based on the same code base but more up to date, is truly the more “mature” code as it has been reviewed, updated, tested, etc. to a greater degree than the early release of the same code.

This is counterintuitive to, say, a car where each release is a fresh thing with new opportunities for mechanical problems and different reliability concerns – where waiting a few years gives you a chance to see what reliability issues get uncovered. Software is not like this. So the concept of wanting more mature software would push you to deploy the “latest and greatest” rather than the “tried and true.”

If we think of software version numbers rather like ages, this comes through.  Linux 3.1 is much older, in terms of software maturing, than Linux 2.4.  It has a decade of additional development.

Let’s use a real world example that is very relevant today. You are in a shop about to install your first server(s). Windows Server 2012 R2 has just released. Should you install Windows Server 2008, 2008 R2 (2010), Server 2012 or Server 2012 R2 (late 2013?)

To many shops, this sounds like we are talking about somewhere between two and four different products entirely which probably have different reasons for choosing each. This, by and large, is untrue. Each newer version is simply an upgrade, update, patch and feature increase on the previous one. Each one, in turn, is more advanced and mature than the one preceding it. Each new version benefits from the work done on the original release of its predecessor as well as bug fixes, patches and feature additions done in the interim between the original release and the successor release. Each new release is, in reality, a “minor release” of the one before it. If we look at the kernel revision numbers, instead of the marketing names of the releases, it might make more sense.

Windows Server 2008 was Windows NT 6.0. Windows Server 2008 R2 was Windows NT 6.1, obviously a minor revision or even a “patch” of the previous release. Windows Server 2012 was Windows NT 6.2 and our current Windows Server 2012 R2 is Windows NT 6.3. If we were to use the revision numbers instead of the marketing names, it sounds almost crazy to intentionally install an old, less mature, less updated and less patched version. We want the latest updates, the latest bug fixes and the latest security issues to have been addressed.

For new software deployments, the newer the software installed, the better opportunity to leverage the latest features and the most time before inevitable obsolescence takes its toll. All software ages so installing newer software gives the best chance that that software will last for the longest time. It provides the best flexibility for the unknown future.

Following this line of thinking might lead us to feel that deploying pre-release or even beta software would make sense as well. And while there might be specific cases where this does make sense, such as in “test groups” to check out software before releasing it to the company at large, in general it does not. The nature of pre-release software is that it is not supported and may contain code which never will be supported. Using such code in isolation can be beneficial, but for general use it is not advised.  There are important processes that are followed between preview or beta releases and final releases of code no matter what maturity level the overall product is at.

That brings us to the other situation, the one in which we are updating existing software. This, of course, is a completely different scenario to a fresh install and there are many, many more factors involved.

One of the biggest factors for most situations is that of licensing. Updating software regularly may incur licensing fees that need to be factored in to the benefits and cost equation. Some products, like most open source software, do not have this cost and can be updated as soon as new versions are available.

The other really large factor in updating software is a human effort cost to updating – unlike in a fresh installation, where the effort of install is effectively a break even between old software and new.  In reality, new software tends to be easier to install than old software simply due to improvements and advancements.  Maintaining a single version of software for a decade means that resources were not dedicated, during that time, to upgrade processes. Upgrading annually during that time means that resources were used ten times to enact separate upgrades. That makes updating much harder to cost justify. But there is more than just the effort of the update process itself, there is also the continuous training needed for end users who will be forced to experience more changes, more often through constant upgrades.

This might make updating software sound like a negative, but it is not. It is simply an equation where each side needs to be weighed. Regular updates often mean small, incremental changes rather than large leaps allowing end users to adapt more naturally. Regular updates mean that update processes are often easier and more predictable. Regular updates mean that technical debt is always managed and the benefits of the newer versions which may be features, efficiencies or security improvements, are available sooner allowing them to be leveraged for a longer period of time.

Taking what we have learned from the two scenarios above, however, there is another important take away to be found here. Once the decision to perform an update has been made, the question is often “to what version do we update?” In reality, however, every update that is more than a standard patching process is really like a miniature “new software” buying decision and the logic as to why we “always” install the newest available version when doing a fresh install also applies here. So when performing an update, we almost always should be updating as far as we can – hopefully to the current version.

To apply the Microsoft example again, we can take an organization that has Windows XP deployed today. The business decides to invest in an update cycle to a newer version, not just continued patching. There are several versions of the Windows desktop platform that are still under active support from Microsoft. These include Windows Vista, Windows 7, Windows 8 and Windows 8.1. Updating to one of the less current versions results in less time before that version’s end of life which increases organizational risk, using older versions means continued investment in already old technologies which means an increase in technical debt and less access to new features which may prove to be beneficial once available. In this particular example, newer versions are also considered to be more secure and require fewer hardware resources.

Every business needs to find the right balance for them for existing software update cycles. Every business and every software package is different. Enterprise software like Microsoft Windows, Microsoft Office or an Oracle Database follow these models very well. Small software projects and those falling near the bespoke range may have a more dynamic and unpredictable release cycle but generally will still follow most of these rules. Consider applying empathy to the software development process to understand how you and your software vendor can best partner to deliver the greatest value to your organization and combine that with your need to reduce technical debt to leverage your software investment in the best possible way for your organization.

But the rules of thumb are relatively easy:

When deploying new or updating, shoot for the latest reasonable version of software.  Use any deployment opportunity to eliminate technical debt as much as possible.

When software already exists weight factors such as human effort, licensing costs, environmental consistency and compatibility testing against benefits in features, performance and technical debt.

The Information Technology Resource for Small Business