Over a period of time, enterprise systems become more tightly knit with increasing interdependency and complexity until, eventually, they collapse under their own mass. Signs of the impending collapse are remarkably similar to black holes - nothing escapes, everything is consumed (people, money and time) and yet no real progress is made.
Worst of all, this appears to be happening with increasing frequency. Rather than determining the root cause and fixing the problem, enterprise systems development seems to have accepted it as a given. Not exactly the pioneering spirit we used to have, where everything was soluble.
Meanwhile, in other universes of endeavour such as car making and film-making, black holes are nowhere to be found. We're constantly amazed at the improvements in these areas, be it increased power and better handling (car making) or more original scripts with better special effects (films/tv, and, by the way, I love "24"). How do these improvements come about and why aren't they happening in enterprise systems development?
[For the purposes of this discussion, I use enterprise systems development as the term for the development of in-house systems in commercial entities be it by in-house or external development staff.]
In my attempt to explain the reasons for the "black hole phenomenon", I'm going to compare the attitudes and approaches of the enterprise systems industry with those of similar creative industries. In particular, I will focus on film-production and car-making.
One thing that stands out immediately in these other industries is a desire to keep things simple. Practitioners actively seek out and remove complexity. The process of shooting a film involves breaking it down into "takes" each of which is one small easy to shoot part of the overall film. Cars are constructed out of collections of components each of which can, to a large extent, be dealt with in isolation from the others. This ruthless application of simplicity is seen as the only way to achieve the end goal, complexity is the enemy. One might reasonably expect to find the same attitude in enterprise systems development but one look at the systems being built and the processes being applied suggests precisely the opposite approach. Complexity is rife, with some practitioners seeming almost to worship it, as if longing to be able to say "I did that". Others justify the whole mess by saying that it's just the nature of software. Why?
One can argue that the development lifetime of a film or car is limited. That is indeed true but, once they're developed, they continue to be useful even if they don't change. In the case of enterprise systems we seem to be unable or unwilling to create the same type of environment. Why?
In film and cars we often see material being archived for future use and reference. These archives are one of a number of tools these industries are creating to assist with educating new recruits. In enterprise systems, there was, for a period of time, an interest in design patterns which encapsulated problem-solving experience and would be continuously added to a library for future reference. This movement, by and large, is isolated these days and given little thought by programmers and architects who seem to be quite happy to make the same mistakes over and over. The enterprise systems industry says it needs more experienced people but seems to lay the responsibility for generating these people at the door of universities and other educational establishments. Why?
Another form of "output" found in these other industries is a constant stream of new experts in the respective arts of car and film creation. The same cannot be said of enterprise systems which seems to rely on external experts in the form of analysts and vendors rather than developing expertise within the people doing the equivalent creative work. Why? [Has it ever occurred to anyone that allowing the vendors to tell their customers what they need is a little backward? Certainly, there are other examples here in the UK, such as mortgage lenders being an important source of information in respect of the health of the housing market - ever noticed how prices are always said to be climbing?]
Each of these other industries has to deal with release schedules but it's interesting to note that the consumers of film and cars don't get to have much influence on how that schedule looks. Essentially, each industry makes those decisions based on a number of factors including:
They aren't always worried about being first to market, they just want a slice of the action. They work to out-innovate each other over a period of time knowing that they are in a war not a battle.
Once again, the world of enterprise systems does things differently. The consumer is allowed a significant input to how long something will take. This creates the potential for those with minimal experience in software creation to have far more influence than they should have. Further, there is a constant undercurrent of do it faster, of being first because, apparently, it's a way of generating critical business advantage. If that were true, surely our other industries would be changing their approach? Are we really saying that the software war is over at the next release?
A key weapon in the armoury of film and car makers is prior planning. In reality, there's considerably more to this planning than just figuring out the schedule. It's research in all sorts of forms - suspension design, location research, storyboarding, prototyping and wind-tunnel testing. These critical elements are all done ahead of the things that require them (so there is room for parallelism given that you can execute on one researched element whilst researching the next) - it is this attention to detail that, for example, allows film-makers to go on site and just put take after take "into the can"- they even rework bits of the storyboard on site but often within the limits of what they've shot already as they can only change the story so much given the boundaries set by previous shooting on a schedule (this is very visible in the production of "24", told you I was a big fan didn't I?). In enterprise systems, all this "planning" is just seen as delay before getting on with the "real" work of coding and testing - how many times has it been said that the upstream elements are critical to success in development and yet these are the first things to be thrown out?
These other industries accept that change is necessary - it is the price of progress towards better special effects or superior car handling or whatever. Meanwhile, enterprise systems development is expecting the equivalent of being able to drive their car to the local dealer and have the entire suspension updated without changing the remainder of the car. To what am I referring? The unrealistic assumption that old tools and platforms should continue to be usable with new development approaches or technologies. The so called preservation of technology investment. We don't expect this from our cars, micro-processors, movie or still cameras, why do we demand it of software tools? I'm not suggesting that enterprises throw all these platforms and tools away, just that these products have a limited useful lifetime which isn't being recognised. After nearly two decades in professional computing, it disappoints me greatly to hear many of today's enterprise programmers complaining when things change or they have to learn something new. Forgive me, but I can remember a time when programmers thrived on this constant change and were actually excited by it. How can one possibly expect progress when everyone resists it so strongly?
[A more insidious inhibitor to change, comes in the form of the adoption of endless "new" technologies that simply re-invent the wheel with nothing going for them other than greater hype than the last effort. Notably, each of these "new" technologies has all the same concepts and abstractions as previous "new" technologies (justified by the fact that it saves developers learning time). If all the same concepts and abstractions are present, the thinking hasn't changed, which means that the approach is broadly the same and no significant progress has been made.]
A fundamental reason for the existence of so many black holes in the enterprise systems development universe is unrealistic expectations. These expectations grow out of the fact that software is seen as being so "malleable". More precisely, I think it's because the effect of such unrealistic expectations doesn't manifest itself in blatant physical terms. Unrealistic expectations in terms of car development would result, for example, in catastrophic suspension failure during testing (or worse, after the car has been sold). In the case of film making, these expectations would likely result in the absence of film footage for editing or a collection of unrelated scenes which could not be edited together.
In reality, software construction, like other creative disciplines, is an exercise in achieving balance between opposing forces:
Whilst other industries accept that a balance must be struck and are considerably more efficient and productive as a result, enterprise systems development has adopted the unrealistic position that it needn't make the necessary trade-offs with costly results. Worse still the analysts and vendors reinforce this behaviour claiming that this or that will make the holy grail of no compromise possible.
Until the people (engineers, customers, managers) involved step back, look at what they are doing with a cold analytical eye and change their behaviour and expectations, things will only get worse with increasing re-implementation costs, creeping complexity, system collapses, staff turnover and a continued stagnation of technological progress where new approaches are introduced and, ultimately, have no effect at all.
Let's hope the necessary changes happen before the entire industry turns into a single, all-consuming black hole.
© Copyright 2003 Dan Creswell