FT : Hiding Complexity
Wednesday, January 28, 2009
Complexity is the enemy of flexibility. It entangles us in unintended consequences. It blocks our attempts to change. It hides potential defects, making it impossible to be sure our systems will function correctly. Performance, transparency, security – all these highly desirable attributes leak away in the face of increasing complexity. Yet one of the tenets of the IT industry has been ever increasing complexity. Moore’s Law, often stated in terms of increasing transistor counts, was originally about complexity: “The complexity for minimum component costs has increased at a rate of roughly a factor of two per year … ” Increasing complexity in the underlying hardware pushes complexity up the stack – more complex operating systems to exploit the hardware, more complex compilers to build applications on top of the operating systems, more complex databases, and so on. If Moore’s Law drives the IT industry, then it’s little wonder we’re enmeshed in complexity. Now, it seems, this complexity is catching up with us. A recent report by PwC (“Why isn’t IT spending creating more value?”), for example, concludes that the growing complexity of large IT systems is limiting our return on investment. While IT investment led to impressive productivity gains in the 1990s, by the early 2000s these gains were being cancelled out by the costs of complexity – increased redundancy, higher maintenance costs, and higher security costs, for example. In the face of this trend, it is easy to fall back to that opening message: keep it simple. It is very easy to look at the IT industry and conclude that it is fatally attracted to complexity. Most engineers are pretty bright people. They can tolerate a lot of complexity and gain a certain type of power by building systems that flaunt it. If only we could get them to focus their intellect instead on eliminating it. The problem with this message is that, for all our best efforts, we almost never eliminate complexity. Most of the time, when we create a system that appears simple, what we have actually done is shift the complexity somewhere else in the technology stack.
Cloud computing is a good example of this. Running an application across a cloud of virtual servers is a lot more complex than running it across a defined number of physical servers in a specific data centre. For a start, there’s an additional, complex piece of software – the hypervisor – that we need to manage. There are also additional complexities in the routing algorithms, authentication protocols, and so on. Cloud computing doesn’t work because it’s simpler than client-server or mainframe computing. It works because we shift the additional complexity to a place where it can be managed more effectively. Companies such as Amazon and Google are simply a lot better at managing servers and operating systems than most other organisations could ever hope to be. By letting Google manage this complexity, an enterprise can then focus more of its own resources on growth and innovation within its core business. Other examples abound. The iPhone has a simple user interface because someone’s done a lot of complex user research and spent a lot of time writing complex user interface code. We pick up our Nintendo Wii’s and start playing with them immediately because large teams spent a lot of time developing complex hardware and software to make them easy to use. In fact, a feature of many of these solutions is that they’ve increased overall complexity. In order to gain simplicity in one dimension, they’ve built additional complexity elsewhere. We shift complexity in other ways too. Paying attention to governance, for example, is hard work, but good governance helps shift complex politics away from operational decision-making, allowing customer-facing teams to focus their energy on satisfying customer needs. Similarly, outsourcing generally succeeds by shifting complexity on to a service provider that has the skills to manage it effectively…..That brings us back to why we need to beware of the easy message – keep it simple. Organisations that try to create fake simplicity generally end up shifting their complexity to some hidden place, where no one can manage it. The more effective strategy is to bring the complexity out into the open, where it can be dealt with. So if we want to continue to gain benefits from IT, we should avoid the siren’s song to reduce complexity. Complexity exists. The best solutions acknowledge that complexity, and shift it to places where it can be managed. Or, in the words of Einstein: “Make everything as simple as possible. But no simpler.”