Architecture, Hyperconvergence, Virtualization

The Commoditization of Architecture

Leave a comment

I often talk about the moving “commodity line”, this line affects essentially all technology, including designs.  Essentially, when any new technology comes out it will start highly proprietary, complex and expensive.  Over time the technology moves towards openness, simplicity and becomes inexpensive.  At some point any given technology becomes goes so far in that direction that it falls over the “commodity” line where it moves from being unique and a differentiator to becoming a commodity and accessible to essentially everyone.

Systems architecture is no different from other technologies in this manner, it is simply a larger, less easily defined topic.  But if we look at systems architecture, especially over the last few decades, we can easily system servers, storage and complete systems moving from the highly proprietary towards the commodity.  Systems were complex and are becoming simple, they were expensive and are becoming inexpensive, they were proprietary and they are becoming open.

Traditionally we dealt with systems that were physical operating systems on bare metal hardware.  But virtualization came along and abstracted this.  Virtualization gave us many of the building blocks for systems commonidization.  Virtualization itself commoditized very quickly and today we have a market flush with free, open and highly enterprise hypervisors and toolsets making virtualization totally commoditized even several years ago.

Storage moved in a similar manner.  First there was independent local storage.  Then the SAN revolution of the 1990s brought us power through storage abstraction and consolidation.  Then the replicated local storage movement moved that complex and expensive abstraction to a more reliable, more open and more simple state.

Now we are witnessing this same movement in the orchestration and management layers of virtualization and storage.  Hyperconvergence is currently taking the majority of systems architectural components and merging them into a cohesive, intelligent singularity that allows for a reduction in human understanding and labour while improving system reliability, durability and performance.  The entirety of the systems architecture space is moving, quite rapidly, toward commoditization.  It is not fully commoditized yet, but the shift is very much in motion.

As in any space, it takes a long time for commoditization to permeate the market.  Just because systems have become commoditized does not mean that non-commodity remnants will not remain in use for a long time to come or that niche proprietary (non-commodity) aspects will not linger on.  Today, for example, systems architecture commoditization is highly limited to the SMB market space as there are effective upper bound limits to hyperconvergence growth that have yet to be tackled, but over time they will be tackled.

What we are witnessing today is a movement from complex to simple within the overall architecture space and we will continue to witness this for several years as the commodity technologies mature, expand, prove themselves, become well known, etc.  The emergence of what we can tell will be commodity technologies has happened but the space has not yet commoditized.  It is an interesting moment where we have what appears to be a very clear vision of the future, some scope in which we can realize its benefits today, a majority of systems and thinking that reside in the legacy proprietary realm and a mostly clear path forward as an industry both in technology focus as well as in education, that will allow us to commoditize more quickly.

Many feel that systems are becoming overly complex, but the opposite is true. Virtualization, modern storage systems, cloud and hyperconverged orchestration layers are all coming together to commoditize first individual architectural components and then architectural design as a whole.  The move towards simplicity, openness and effectiveness is happening, is visible and is moving at a very healthy pace.  The future of systems architecture is one that clearly is going to free IT professionals from spending so much time thinking about systems design and more time thinking about how to drive competitive advantage to their individual organizations.

Business of IT, Career

No One Ever Got Fired For Buying…

Leave a comment

It was the 1980s when I first heard this phrase in IT and it was “no one ever got fired for buying IBM.”  The idea was that IBM was so well known, trusted and reliable that it was the safe choice as a vendor for a technology decision maker to select.  As long as you chose IBM, you were not going to get in trouble, no matter how costly or effective the resulting solution turned out to be.

The statement on its own feels like a simple one.  It makes for an excellent marketing message and IBM, understandably, loved it.  But it is what is implied by the message that causes so much concern.

First, we need to understand what the role of the IT decision maker in question is.  This might sound simple, but it is surprising how easily it can be overlooked.  Once we delve into the ramifications of the statement itself, it is far too easy to lose track of the real goals. In the role of a decision maker, the IT professional is tasked with selecting the best solution for their organization based on its ability to meet organizational goals (normally profits).  This means evaluating options, shielding non-technical management from sales people and marketing, understanding the marketplace, research and careful evaluation.  These things seem obvious, until we begin to put things into practice.

What we have to then analyze is not that “no one ever got fired for choosing product X”, but what the ramifications of such a statement actually are.

First, the statement implies an organization that is going to judge IT decision making not on its merits or applicability but on the brand name recognition of the product maker.  In order for a statement like this to have any truth behind it, it requires the entire organization to either lack the ability or desire to evaluate decisions but also an organizational desire to see large, expensive brand names (the statement is always made in conjunction with extremely high cost items compared to the alternatives) over other alternatives.  An organizational preference towards expensive, harder to justify spends is a dangerous one at best.  We assume that not only does buying the most expensive, most famous products will be judged well compared to less expensive or less well known ones, but that buying products is seen as beneficial to not buying products; even though often the best IT decisions are to not buy things when no need exists.  Prioritizing spending over savings for their own reasons without consideration for the business need is very bad, indeed.

Second, now that we realize the organizational reality that this implies, that the IT decision maker is willing to seize this opportunity to leverage corporate politics as a means of avoiding taking the time and effort to make a true assessment of needs for the business but rather skip this process, possibly completely, we have a strong question of ethics.  Essentially, whether out of fear of the organization not properly evaluating the results  or by blaming the decision maker for unforeseeable events after the fact or of looking to take advantage of the situation to be paid for a job that was not done, we have a significant problem either individually, organizationally, or both.

For any IT decision maker to use this mindset, one that there is safety in a given decision regardless of suitability, there has to be a fundamental distrust of the organization.  Whether this is true of the organization or not is not known, but that the IT decision maker believes it is required for such a thought to even exist.  In many organizations it is understandable that politics trump good decision making and it is far more important to make decisions for which you cannot be blamed rather than trying honestly to do a good job.  That is sad enough on its own, but so often it is simply an opportunity to skip the very job for which the IT decision maker is hired and paid and instead of doing a difficult job that requires deep business and technical knowledge, market research, cost analysis and more – simply allowing a vendor to sell whatever they want to the business.

At best, it would seem, we have an IT decision maker with little to no faith in the ethics or capabilities of those above them in the organization.  At worst we have someone actively attempting to take advantage of a business by being paid to be a key decision maker while, instead of doing the job for which they are hired or even doing nothing at all, actively putting their weight behind a vendor that was not properly evaluated based possibly solely on not needing to do any of the work themselves.

What should worry an organization is not that vendors that could often be considered “safe” get recommended or selected, but rather why they were selected.  Vendors that fall into this category often offer many great products and solutions or they would not earn this reputation in the first place.  But likewise, after gaining such a reputation those same vendors have a strong financial incentive to take advantage of this culture and charge more while delivering less as they are not being selected, in many cases, on their merits but instead on their name, reputation or marketing prowess.

How does an organization address this effect?  There are two ways.  One is to evaluate all decisions carefully in a post mortem structure to understand what good decisions look like and not limit post mortems to obviously failed projects.  The second is to look more critical, rather than less critically, at popular product and solution decisions as these are red flags that decision making may be being skipped or undertaken with less than the appropriate rigor.  Popular companies, assumed standard approaches, solutions found commonly in advertising or commonly recommended by sales people, resellers, and vendors should be looked at with a discerning eye, moreso than less common, more politically “risky” choices.

 

Storage

SMBs Must Stop Looking to BackBlaze for Guidance

Leave a comment

I have to preface this article, because people often take these things out of context and react strongly to things that were never said, with the disclaimer that I think that BackBlaze does a great job, has brilliant people working for them and has done an awesome job of designing and leveraging technology that is absolutely applicable and appropriate for their needs.  Nothing, and I truly mean nothing, in this article is ever to be taken out of context and stated as a negative about BackBlaze.  If anything in this article appears or feels to state otherwise, please reread and confirm that such was said and, if so, inform me so that I may correct it.  There is no intention of this article to imply, in any way whatsoever, that BackBlaze is not doing what is smart for them, their business and their customers.  Now on to the article:

I have found over the past many years that many small and medium business IT professionals have become enamored by what they see as a miracle of low cost, high capacity storage in what is know as the BackBlaze POD design.  Essentially the BackBlaze POD is a low cost, high capacity, low performance nearly whitebox storage server built from a custom chassis and consumer parts to make a disposable storage node used in large storage RAIN arrays leveraging erasure encoding.  BackBlaze custom designed the original POD, and released its design to the public, for exclusive use in their large scale hosted backup datacenters where the PODs functions as individual nodes within a massive array of nodes with replication between them.  Over the years, BackBlaze has updated its POD design as technology has changed and issues have been addressed.  But the fundamental use case has remained the same.

I have to compare this to the SAM-SD approach to storage which follows a similar tact but does so using enterprise grade, supported hardware.  These differences sometimes come off as trivial, but they are anything but trivial, they are key underpinnings to what makes the different solutions appropriate in different situations.  The idea behind the SAM-SD is that storage needs to be reliable and designed from the hardware up to be as reliable as possible and well supported for when things fail.  The POD takes the opposite approach making the individual server unreliable and ephemeral in nature and designed to be disposed of rather than repaired at all.  The SAM-SD design assumes that the individual server is important, even critical – anything but disposable.

The SAM-SD concept, which is literally nothing more than an approach to building open storage, is designed with the SMB storage market in mind.  The BackBlaze POD is designed with an extremely niche, large scale, special case consumer backup market in mind.  The SAM-SD is meant to be run by small businesses, even those without internal IT.  The POD is designed to be deployed and managed by a full time, dedicated storage engineering team.

Because the BackBlaze POD is designed by experts, for experts in the very largest of storage situations it can be confusing and easily misunderstood by non-storage experts in the small business space.  In fact, it is so often misunderstood that objections to it are often met with “I think BackBlaze knows what they are doing” comments, which demonstrates the extreme misunderstanding that exists with the approach.  Of course BackBlaze knows what they are doing, but they are not doing what any SMB is doing.

The release of the POD design to the public causes much confusion because it is only one part of a greater whole.  The design of the full data center and the means of redundancy and mechanisms for such between the PODs is not public, but is proprietary.  So the POD itself represents only a single node of a cluster (or Vault) and does not reflect the clustering itself, which is where the most important work takes place.  In fact the POD design itself is nothing more than the work done by the Sun Thumper and SAM-SD projects of the past decade without the constraints of reliability.  The POD should not be a novel design, but an obvious one.  One that has for decades been avoided in the SMB storage space because it is so dramatically non-applicable.

Because the clustering and replication aspects are ignored when talking about the BackBlaze POD some huge assumptions tend to be made about the capacity of a POD that has much lower overhead than BackBlaze themselves get for the POD infrastructure, even at scale.  For example, in RAID terms, this would be similar to assuming that the POD is RAID 6 (with only 5% overhead) because that is the RAID of an individual component when, in fact, RAID 61 ( 55% overhead) is used!  In fact, many SMB IT Professionals when looking to use a POD design actually consider simply using RAID 6 in addition to only using a single POD.  The degree to which this does not follow BackBlaze’s model is staggering.

BackBlaze: “Backblaze Vaults combine twenty physical Storage Pods into one virtual chassis. The Vault software implements our own Reed-Solomon encoding to spread data shards across all twenty pods in the Vault simultaneously, dramatically improving durability.”

To make the POD a consideration for the SMB market it is required that the entire concept of the POD be taken completely out of context.  Both its intended use case and its implementation.  What makes BackBlaze special is totally removed and only the most trivial, cursory aspects are taken and turned into something that in no way resembles the BackBlaze vision or purpose.

Digging into where the BackBlaze POD is differing in design from the standard needs of a normal business we find these problems:

  • The POD is designed to be unreliable, to rely upon a reliability and replication layer at the super-POD level that requires a large number of PODs to be deployed and for data to be redundant between them by means of custom replication or clustering.  Without this layer, the POD is completely out of context.  The super-POD level is known internally as the BackBlaze Vault.
  • The POD is designed to be deployed in an enterprise datacenter with careful vibration dampening, power conditioning and environmental systems.  It is less resilient to these issues as standard enterprise hardware.
  • The POD is designed to typically be replaced as a complete unit rather than repairing a node in situ.  This is the opposite of standard enterprise hardware with hot swap components designed to be repaired without interruption, let alone without full replacement.  We call this a disposable or ephemeral use case.
  • The POD is designed to be incredibly low cost for very slow, cold storage needs.  While this can exist in an SMB, typically it does not.
  • The POD is designed to be a single, high capacity storage node in a pool of insanely large capacity.  Few SMBs can leverage even the storage potential of a single POD let alone a pool large enough to justify the POD design.
  • The BackBlaze POD is designed to use custom erasure encoding, not RAID.  RAID is not effective at this scale even at the single POD level.
  • An individual POD is designed for 180TB of capacity and a Vault scale of 3.6PB.

Current reference of the BackBlaze POD 5: https://www.backblaze.com/blog/cloud-storage-hardware/

In short, the BackBlaze POD is a brilliant underpinning to a brilliant service that meets a very specific need that is as far removed from the needs of the SMB storage market as one can reasonably be.  Respect BackBlaze for their great work, but do not try to emulate it.

 

Career, Education

Choosing a University for IT Education

Leave a comment

In previous articles I have tackled the questions around approaching university education and selecting a degree program but, thus far, I have not provided any guidance in selecting an institution at which to study.  That will be rectified now.

There are basically five categories of universities in the United States that we need to consider.  These types of educational institutions are:

  • Unaccredited schools
  • Accredited Trade schools
  • Accredited Online schools
  • Accredited Brick and Mortar Private schools
  • Public Brick and Mortal schools

There are more types of schools than these but we can basically lump all schools into one of these categories as these are the general categories that a hiring manager will view schools on a candidate’s resume.  University education has two key benefits, the first is in broadening thought processes and introducing students to many topics through liberal studies.  The second is in providing beneficial resume line items and for this second category we need a university that provides a positive reaction.

So assuming that we are concerned about putting our degrees and education onto our resumes, we need to consider carefully how our choices of educational institution will reflect on us.  You will notice that I carefully did not say that universities provide skill training to prepare workers for the jobs that they will do.  This I have covered in other articles; the university system is not intended nor generally capable of training people directly for work.  There is no mandate to do this, no expectation and little potential capacity especially when we are considering highly technical or quickly changing career fields.  IT maybe be among the most extreme of these kinds of fields, but this issue applies across the board.

Because such a huge portion of the value of a degree comes from how that degree is perceived by a hiring manager, we have to consider that impression very carefully.  And this produces what I would consider “the dead line” in selecting educational institutions.

For a large percentage of hiring managers, and much of the population, only certain types of universities are considered valid.  This is not a judgment call, only an observation of hiring reality.  Whether the quality of education, rigors of study and such are valuable or not, certain categories of schools are considered non-valid in enough of the marketplace that we must effectively discount them from consideration.

From the list that I have provided, any school that is unaccredited, purely online or a tech/trade school should be completely avoided.  These three categories are routinely views as such a strong negative that in a great many cases a candidate will be eliminated based on this one factor alone.  It is commonly said that hiring managers will see one of these schools and throw a resume directly out without any further consideration, but in reality in many cases an HR filter will do this before any human even sees the resume.  The same logic that says that we use degrees to get passed human resource gatekeepers to get our resumes in front of hiring managers based on “black and white” filter requirements, also tells us that we must avoid schools that would be considered to be on a “black list.”

This leaves only two categories of schools for any serious consideration: private, accredited brick and mortar schools and public, accredited brick and mortar schools.  Now, it must be noted, that just because a school is brick and mortar does not mean that they do not also offer online or alternative classes.  And at no point has it been suggested that it is necessary to attend a school in person.  What is critical is simply that the school be perceived as a valid, traditional educational institution.  In many cases, online classes are the best option as they provide more flexibility and better use of time avoiding time wasted in commuting, moving between classrooms and such.

Of this remaining category, public schools fare far better than private ones because the lower cost of attendance lowest, quite dramatically, the risk inherent in spending time and money on education: the less money spent, the less risk taken.  In only rare cases are private schools any better than public ones and in very many cases, they are worse.  The risk/reward calculation on most public schools is simply far better in the majority of cases.

With any school choice, reputation matters.  Schools with a good reputation are best, especially those that are broadly known.  Schools that have no reputation can be fine, as long as they truly are unknown and fall into good categories.  Schools can get a bad reputation regionally or globally, however, and this poses a risk that is difficult to predict or to avoid.  What is a top ranked school today can be poorly viewed tomorrow, and vice versa.  Large schools have the advantage of increasing the chances that someone on a hiring team will have attended that school increasing personal affinity.

There is no simple answer to selecting the right school.  Does the school benefit you through education, reputation or association (with people that will help you later in your career) is unique to each person and school combination.  But he universal guideline to follow is to stick to accredited, broadly well respected, brick and mortar, public or private not for profit schools and consider cost carefully.  Avoid online and/or for profit schools or any school that lacks proper accreditation.

As a modern side note: many schools, even sometimes others good ones, that advertise heavily especially on television or radio, often earn a bad reputation simply because of the medium of attempting to lure students.  If you have seen a school because of their marketing campaign, assume that a hiring manager has as well and while some good schools do this, it may not matter.

The Information Technology Resource for Small Business