Pragmatic Dictator

A graphic demonstration of how short-term thinking over strategy can be outright life-threatening:

I was out driving this morning on a dual carriageway, approaching a slip-road in lane two (the outside lane) so as to avoid merging traffic. In front of me is another driver who in their impatience is on the boot-lid of another vehicle, we’re right on top of the merge point for the slip-road now which is a feeder from the M4. At this point, the boot-lid surfer dives into the inside lane clearly intent on an undertake, failing to account for the motor-cycllst merging from the slip road. What follows is best described as a lucky escape with both motor-cyclist and boot-lid surfer taking evasive action.

Many a technical decision is taken without measuring, pausing for strategic thought or evaluating alternatives. What we typically do is jump on whatever is the current hype without bothering to weigh up whether it’s worth it. This is a key contributory factor to the utter absurdity that is the hype curve where we all get far too optimistic about a technology, jump on it and then after a few years realise it is not the holy grail. Techies as a group are supposed to be some of the most rational, analytical of all and yet we make this newbie mistake – who are we kidding?

Short-term thinking can be seen elsewhere too such as in the feature’s versus refactoring/re-structuring discussions that go on in software product (shrink-wrap, web or otherwise) companies across the world. In general it seems that businesses encourage these minimal thought policies. Increase customers now. Increase revenue now. Don’t worry about consequences we’ll deal with those later………maybe!

But wait a minute – if a child wants to stick their fingers in the wall-socket should we just forget the consequences and let them do it? Clearly not – that’s strategic, big picture thinking which saved our child some pain. Why don’t we do similar things and save our businesses, systems and workers pain?

I do wonder if rampant short-termism is somewhat related to our own survival instincts. Perhaps when there’s too much pressure/threat we are more likely to stop thinking about the big picture? Understandable when it’s life or death but surely unnecessary in business, tech or the daily commute?

Technorati Tags: business, strategy, technology

I see many a comment talking about concerns with roundtrips in SOA/distributed architectures. For example in “The Fractal Nature of Web Services” the author starts totalling up additional round trips taken once we disassemble what was a single app into separate remote services. He then goes on to walk down the well worn performance path expressing concern about performance etc.

His concern is certainly valid but it makes a fundamental assumption about how you implement your services:

Each service is represented by some stub that communicates via some protocol to some remote back-end. Clients access the service via the stub and thus incur remote call costs.

Further whether he meant it or not, the author gives the impression that SOA can only be done with Web Services which is absolutely not true. Your choice of method for implementation for SOA is determined by your chosen architectural constraints and can often (and maybe, should always) lead to a hybrid model with “public” interfaces exposed via WS-* or REST and others implemented in Java.

And this is where it gets interesting because with Java we can break the stub/remote back-end constraint. Why? Because we have code-downloading. We can construct a service implementation that offers an interface and is advertised in a service registry as usual but at the point of access, downloads it’s whole self to the client. The service then runs in the client’s address space.

i.e. We have what appears to be a remote service with none of the roundtrip costs because in fact it runs locally.

From a practical perspective that means we can do things like:

1. Create a service that wraps a database and contains all the SQL etc
2. Advertise that service
3. Client downloads service and all it’s dependent code and configuration
4. Client invokes on service
5. SQL connection is made to database
6. Queries are run against database

Now, were the service genuinely remote we would incur two roundtrips:

1. Client to Service
2. Service to Database

But because we can “inject” the service into the client we actually only have the roundtrip to the database. Things to note:

• We keep all the benefits of late binding of services and clients at runtime.
• We taught the client a new protocol on the fly – we downloaded all the client/database communication code at runtime as part of service lookup.

Technorati Tags: distributed systems, Java, soa

Argh, there it is again – the DVLA TV advert that warns of computers constantly trawling for those that haven’t paid their car tax and sending automatic fines. It comes complete with menacing and threatening imagery all in black, white and grey.

Then there’s the railway stations – on my daily journey through Reading and Paddington I’m assaulted with automated brainwash messages about security, late trains, slippery surfaces, surveillance etc.

Has anyone noticed how 1984 this all is? Do we think this is at all a good thing? What does it say about day to day life in our “modern” society? What might it being doing to our creativity, our spirit?

This stuff needs to stop…….

Technorati Tags: personal, philosophy

Normally we class availability as a non-functional requirement of the systems we build but I’m starting to think that’s a mistake because often when we get to project scheduling it’s forgotten. We allocate time, budget and resource to make sure we’re feature complete, that this button or that tick box is visible or that we can gather credit card details but rarely do we treat availability similarly.

Unlike a functional requirement which has some form of physical manifestation availability tends to be invisible dotted around the system in all sorts of different places from architecture and code to hardware. In contrast real features are a logical sequence of inter-linked pieces of code and design that are naturally traceable from entry to exit point. No entry point means the feature is not complete and this can be trivially determined prior to deployment. The first time we know availability isn’t “feature complete” is when our deployed website is down.

Worse, with every single feature or subsystem we add and the corresponding increase in load and size of dependency graph there’s an increased chance that we compromise availability. This can make for some serious growing pains where we add more features, attract more customers and thus more load all the time paying no attention to availability until it’s too late.

Consider the wiring in a house, we add a garage which needs lighting, maybe heating and a motor-powered door. Then we build an extension for guests, more lighting, heating and wall sockets. Eventually, something gives, maybe it’s a piece of wiring that should’ve been replaced or a bad junction but the net effect is to risk burning out all the wiring in the house. Replacing all the wiring would’ve been hard enough without the extensions but now the house is twice as big and the wiring is four times more complex. Worse, we cannot do a piecemeal upgrade instead we must replace all the wiring at once and until it’s done we can’t have all the heating on or we can’t cook at the same time as using the garage.

Like the wiring, availability needs constant and sometimes significant attention and putting the work off can lead to significant cost down the line. Similar arguments apply to other non-functional aspects of our systems such as scalability (it’s no good adding features that attract more customers if the system won’t cope with the additional load).

In light of the above, I believe it might be better to classify availability as a feature if not the feature. After all if the system is not available, none of the cool features we’ve implemented are worth anything because they can’t be accessed. It might also be a catalyst for discussion (especially amongst non-technical staff) of availability’s costs and benefits as compared to those of other features leading to improved strategic thinking.

Relevant Links:

• Vogels on Scalability
• SD Forum 2006 – The eBay architecture
• Inside MySpace.com
• Vogels – E-Commerce at Interplanetary Scale
• A Conversation With Werner Vogels

Technorati Tags: architecture, distributed systems, availability, web

I came across a new concept in my travels this week, an invitation with compulsory acceptance.

Is an invitation that is compulsory to accept an invitation at all? Surely most people would see it as nothing more than a dressed up way of demanding attendance? Surely such “garnish” (Ben Elton fan’s will know exactly what I’m talking about) just engenders bad feeling in many recipients?

Meanwhile in the world of systems…….

We often do the equivalent of compulsory invitations in the way we build software – consider for example the whole dependency injection thing.

In many cases we build some object via a constructor that expects to be injected with a whole heap of other things. How do we guarantee all those things will attend and remain present for the lifetime of our object? After all without these things our object cannot perform it’s intended task. About the only way this works is by insisting that all these things exist within the same JVM as the object to be injected – then they all either exist or don’t exist.

Clearly, this doesn’t work so well in distributed systems where achieving the same availability assurance is considerably more difficult. A slightly more subtle issue is that if our systems are written in the above manner, upgrades are also more difficult because we must manage a dependency tree. e.g. We wish to upgrade System A and it has dependants, Systems B and C both of which will need to be taken off-line whilst we upgrade A.

What is required is a more dynamic form of injection, something that can be reinitialised or changed which at least invalidates injection via constructors (well, unless we throw the whole object out and rebuild it from scratch). In fact, we probably require some kind of event-based solution such that systems can get liveness information about the systems they rely upon and take appropriate action (which will likely include dealing with in flight operations).

Technorati Tags: architecture, design, distributed systems, philosophy