Tag Archives: planning

What Do I Do Now? Planning for Design Changes

Quite often I am faced with talking to people about their system designs, plans and architectures.  And many times that discussion happens too late and designs are either already implemented or they are partially implemented.  This can be very frustrating if the design in progress has been deemed to not be ideal for the situation.

I understand the feeling of frustration that will come from a situation like this but it is something that we in IT must face on a very regular basis and managing this reaction constructively is a key IT skill.  We must become masters of this situation both technically as well as emotionally.  We should not be crippled by it, it is a natural situation that every IT professional will experience on a regular basis.  It should not be discouraging or crippling but it is very understandable that it can feel that way.

One key reason that we experience this so often is because IT is a massive field with a great number of variables to be considered in every situation.  It is also a highly creative field where there can be numerous, viable approaches to any given problem.  That there is even a single “best” option is rarely true.  Normally there any many competitive options.  Sometimes these are very closely related, sometimes these options are drastically different making them very difficult to compare meaningfully.

Another key reason is that factors change.  This could be that new techniques or information come to light, new products are released, products are updated, prices change or business needs change near to or even during the decision making and design processes.  This rate of change is not something that we, as IT professionals, can hope to ever control.  It is something that we must accept and deal with as best as we can.

Another thing that I often see missed is that a solution that was ideal when made may not be ideal if the same decision was being made today.  This does not, in any way, constitute a deficiency in the original design yet I have seen many people react to it as if it did.  The most common scenario that I run into where I see people exhibit this behaviour is in the aversion to the use of RAID 5 in modern storage design, RAID 6 and RAID 10 being the popular alternatives for good reason.  But this RAID 5 aversion, common since about 2009, did not exist always and from the middle of the 1990s until nearly the end of the 2000s RAID 5 was not only viable, it was very commonly the best solution for the given business and technical needs (the increase in aversion to it was mostly gradual, not sudden.)  However many people see RAID 5 as understandably poor as an option today but apply this new aversion to systems designed and implemented long ago, sometimes close to two decades ago.  This makes no sense and is purely an emotional reaction.  RAID 5 being the best choice for a scenario in 2002 in no way implies that it will still be the best choice in 2015.  But likewise, RAID 5 being a poor choice in 2015 for a scenario in no way belittles or negates the fact that it was very often a great choice several years ago.

I have been asked many times what to do once less than ideal design decisions have been made.  “What do I do now?”

Learning what to do when perfection is no longer an option (as if it ever really was, all IT is about compromises) is a very important skill.  The first things that we must tackle are the emotional problems as these will undermine everything else.  We must do our best to step back, accept the situation and act rationally.  The last thing that we want to do is take a non-ideal situation and make things worse by attempting to reverse justify bad decisions or panicking.

Accepting that no design is perfect, that there is no way to always get things completely right and that dealing with this is just part of working IT is the first step.  Step back, breathe deep.  It isn’t that bad.  This is not a unique situation.  Every IT pro doing design goes through this all of the time.  You should try your best to make the best decisions possible but you must also accept that that can rarely be done – no one has access to enough resources to really be able to do that.  We work with what we have.  So here we are.  What’s next?

Next is to assess the situation.  Where are we now?  In many cases the implementation is done and there is nothing more to do.  The situation is not ideal, but is it bad?  Very often the biggest mistake that I see people facing of an all ready implemented design is that it is too costly – typically “better” solutions are not better because they are faster or more reliable but are better because they are cheaper, easier or faster to have implemented.  That’s an unfortunate situation but hardly a crippling one.  Whatever time or money was spent must have been an acceptable amount at the time and must have been approved.  The best that we can do, right now, is learn from the decision process and attempt to avoid the overspending in the future.  It does not mean that the existing solution will not work or even not work amazingly well.  It is simply that it may not have been a perfect choice given the business needs, primarily financial, involved.

There are situations where a design that has been implemented does not adequately meet the stated business requirements.  This is thankfully less common, in my experience, as it is a much more difficult situation.  In this case we need to make some modifications in order to fulfill our business needs.  This may prove to be expensive or complex.  But things may not be as bad as what they seem.  Often reactions to this are misleading and the situation can often be salvaged.

The first step once we are in a position where we have implemented a solution that fails to meet business needs is to reassess the business needs.  This is not to imply that we should fudge the needs to massage them into being whatever our system is able to fulfill, not at all.  But it is a good time to go back and see if the originally stated needs are truly valid or if they were simply not vetted well enough or, even more likely, to go and see if the business needs have changed during the time that the implementation took place.  It may be that the implemented solution does, in fact, meet actual business needs even if they were originally misstated or because the needs have changed over time.   Or it might be that business needs have changed so dramatically that even perfect planning would originally have fallen short of the existing needs and the fact that the implemented solution does not perform as expected is of minor consequence.   I have been very surprised just how often this verification of business needs has turned a solution believed to be inadequate into an “overkill” solution that actually cost more than necessary simply because no one pushed back on overstatements of business needs or no one questioned financial value to certain technology investments.

The second step is to create a new technology baseline.  This is a very important step to assist in preventing IT from falling into the drop of the sunk cost fallacy.  It is extremely common for anyone, this is not unique to IT in any way, to look at the time and money spent on a project and assume that continuing down the original path, no matter how foolish it is, is the way to go because so many resources have been expended on that path already.  But this makes no sense, of course, how you got to your current state is irrelevant.  What is relevant is assessing the current needs of the department and company and taking stock of the currently available solutions, technologies and resources.  Given the current state, the best course forward can be determined.  Any consideration given to the effort expended to get to the current state is only misleading.

A good example of the sunk cost fallacy is in the game of chess.  With each move it is important to assess all available moves, risks and strategies again because what moves were used to get to the current state have no bearing on what moves make sense going forward.  If the world’s greatest chess player or an amazing computer chess algorithm was to be brought in mid-game they would not require any knowledge as to how the current state had come to be – they would simply assess the current state and create a strategy based upon it.

This is the same as we should be behaving in IT.  Our current state is our current state.  It does not matter for strategic planning what unfolded to get us into that state.  We only care about those decisions and costs when doing a post mortem process in order to determine where decision making may have failed in order to learn from it.  Learning about ourselves and our processes is very important.  But that is a very different task from doing strategic planning for the current initiative.

The unfortunate thing here is that we must begin our planning processes again but this time with, we assume, more to work with.    But this cannot be avoided.  In the worst cases, budgets are no longer available and there are no resources to fix the flawed design and achieve the necessary business goals.  Compromises sometimes are necessary.  Making do with what we have is sometimes that best that we can do. But, in the vast majority of cases it would seem, some combination of additional budget or creative reuse of existing products can be adequate to remedy the situation.

Once we have reached a state in which we have addressed our short falls, whether simply by accepting that we have over spent, under-delivered or have adjusted to meet needs we have an opportunity to go back and investigate our decision making processes.  It is by doing this that we hope to grow as individuals and, if at all possible, on an organizational level to learn from our mistakes, or determine if there even were mistakes.  Every company and every individual makes mistakes.  What separates us is the ability to learn from them and avoid those same mistakes in the future.  Growth comes primarily from experiencing pain in this way and while often unpleasant to face it is here that we have the best opportunity to create real, lasting value.  Do not push off or skip this opportunity for review whether it be a harsh, personal review that you do yourself or a formal, organizational review run by people trained to do so or something in-between.  The sooner that the decision processes are evaluated the fresher the memory will be and the sooner the course correction can take effect.

The final step that we can do is to begin the decision process to design a replacement for the current implementation as soon as possible, once the review of the decision process is complete.  This does not necessarily mean that we should intend to spend money or change our designs in the near future.  Not at all.  But by being extremely pro-active in design making we can attempt to avoid the problems of the past by giving ourselves additional time for planning, more time for requirements gathering and documentation, better insight into changes in requirements over time by regularly revisiting those requirements to see if they remain stable or if they are changing, more opportunity to get management and peer buy in and investment in the decision and better understanding of the problem domain so that we are better equipped to alter the intended design or know when to scrap it and start over before implementing it the next time.  It also could, potentially, give us a better chance of codifying organizational knowledge that could be passed on to a successor should you yourself not be in the position of decision making or implementation when the next cycle comes around.

With good, rational processes and a good understanding of the steps that need to be taken in a case of less than ideal systems design or implementation we can recover from missteps and not only, in most cases, recover from them in the short term but we can insulate the organization from the same mistakes in the future.

You Aren’t Gonna Need It

I’m lucky that I work in IT but come from a software engineering background, this gives me a bit of a different perspective on the world of IT both in understanding much of what is happening behind the scenes with release cycles and features and also in applying knowledge gained from that industry to this one.

In the software engineering community in recent years the concept of “You Aren’t Gonna Need It” or YAGNI has become a popular one.  YAGNI arose from the Extreme Programming (XP) group of Agile developers and is stated as this rule: “Always implement things when you actually need them, never when you just foresee that you need them.”

I like to rephrase YAGNI in development to “Don’t invest in something until you know that you need it.”  But the concept is the same – if you spend time and money building pieces that you aren’t sure that you will ever need you take on risks such as not getting value as early as possible (by focusing on the things that don’t matter yet while neglecting the things that do) and investing in technology that will never be used (because requirements change, project gets canceled, etc.)

This concept ports over to IT extremely well.  Design and purchasing are both heavily influences, or should be, by YAGNI.  Storage is a great example.  Don’t invest in storage today that you think that you will use tomorrow.  We can list a lot of reasons for why early storage investment is bad: the business has little to no ability to accurately predict its own growth, IT is poor at predicting storage growth based on business growth, the time-value of money and buying storage today is more costly than buying the same storage tomorrow.  Anytime that we buy based on predictions we take on risk.  Predictions rarely come true.

If we over buy storage today we are paying a premium for that storage because storage costs drop dramatically over time.  If we buy with 100% headroom and it takes three years or more before we use that headroom we are paying too much for the storage and getting older technology when buying later would give us better insight into what we actually need at that time (not just capacity but speed, reliability, features, etc.), lower cost and more options.

Overbuying is one risk, underbuying is another.  Underbuying is, obviously, less of a risk but still a concern.  If you buy today for needs three years out and at two years suddenly have a burst of need you may have overinvested in a platform or technology that cannot meet your needs.

Storage is one example but this can apply anywhere from software licenses, to CPU capacity, memory, high availability technologies even desktops.  Few shops would over buy desktops by a hundred percent just to be ready for a predicted head count increase in three years, but strangely they won’t hesitate doing it elsewhere.

By buying what is required for the immediate need and holding purchasing decisions until later there is a significant opportunity for cost savings and technology improvements.  In some cases it may be that the future need never arises whether because of bad predictions, changes in market or strategy or a change in technology direction either internally or externally.

Beyond purchasing, YAGNI can apply to network design.  It is not uncommon for large, complex designs to be proposed and implemented based on anticipated growth often years away and, to be honest, seldom very likely in a realistic world.  Building, for example, a complex high availability environment with expensive licensing, complex networking, lots of storage for an expected company growth in the future when just two servers and a nice backup plan is all that is cost justified today is dangerous.  Not only must the necessary growth happen to justify the IT spend but it must happen so quickly that the time-value of the money is justified and the cost of the technology does not drop so much as to have made implementing two systems more cost effective.  It is surprising how easily it can happen that putting in a smaller, stop-gap system and then implementing a larger scale system when needed can be far cheaper just because the cost of building the larger, more complex system has dropped in price so much since the first system was put in place and that is before taking into account the risk of bad predictions.

Spending early has an additional risk – it ties up corporate finances in unused architecture.  That money could be invested in other parts of the business in order to grow the business.  In extreme cases, overinvestment in infrastructure could be a contributor to a company failing completely – a self fulfilling situation where not using YAGNI in and of itself created the situation where YAGNI most applied.  The architected solution was never needed as the company failed.

YAGNI is a risk mitigation process.  Working with the needs that you know versus the needs that you anticipate.

Maybe IT shops over buy today because they are given specific budgets.  This is understandable that IT ends up in a technology grab attempting to implement whatever they can when the whims of the business smile upon them.  This, however, is an extremely poor business practice.  Businesses need to realize that large sums of money are being wasted on IT because IT is forced to implement systems with the assumption of clairvoyancey based on arbitrary budgets from the business with no basis in the real world.  IT is stuck buying what they can “sell” to the business based on often very unclear factors and the business often funds IT quite capriciously.  The creates a very unhealthy business and IT relationship where IT is wasting money because it has little choice and the business sees IT as a waste because they are not allowed to operate efficiently.

To fix this situation the business and IT need to work together.  IT needs to act more like a business-savvy unit and the business needs to lean on IT for guidance and not use prediction-based budgeting or become entangled in picking technological approaches without the technical understanding of the ramifications of those choices.  IT needs to be able to trust the business to make logical business financial decisions and the business needs to be able to trust IT to make logical technological decisions for the business.  The business drives IT, IT enables the business.  It is a symbiotic relationship.  If the business insists on making IT predict and operate on fixed budgets, IT will continue to be forced to overspend and overarchitect whenever possible in the hopes of being prepared for tomorrow when the budget may not be approved.  If IT was trusted to request what is needed and the business was trusted to fund technology needs at the appropriate time both could operate more effectively for the common good.

Takeaway: Don’t invest early, you don’t know what technology or the business will do tomorrow.