A graphic demonstration of how short-term thinking over strategy can be outright life-threatening:

I was out driving this morning on a dual carriageway, approaching a slip-road in lane two (the outside lane) so as to avoid merging traffic. In front of me is another driver who in their impatience is on the boot-lid of another vehicle, we’re right on top of the merge point for the slip-road now which is a feeder from the M4. At this point, the boot-lid surfer dives into the inside lane clearly intent on an undertake, failing to account for the motor-cycllst merging from the slip road. What follows is best described as a lucky escape with both motor-cyclist and boot-lid surfer taking evasive action.

Many a technical decision is taken without measuring, pausing for strategic thought or evaluating alternatives. What we typically do is jump on whatever is the current hype without bothering to weigh up whether it’s worth it. This is a key contributory factor to the utter absurdity that is the hype curve where we all get far too optimistic about a technology, jump on it and then after a few years realise it is not the holy grail. Techies as a group are supposed to be some of the most rational, analytical of all and yet we make this newbie mistake - who are we kidding?

Short-term thinking can be seen elsewhere too such as in the feature’s versus refactoring/re-structuring discussions that go on in software product (shrink-wrap, web or otherwise) companies across the world. In general it seems that businesses encourage these minimal thought policies. Increase customers now. Increase revenue now. Don’t worry about consequences we’ll deal with those later………maybe!

But wait a minute - if a child wants to stick their fingers in the wall-socket should we just forget the consequences and let them do it? Clearly not - that’s strategic, big picture thinking which saved our child some pain. Why don’t we do similar things and save our businesses, systems and workers pain?

I do wonder if rampant short-termism is somewhat related to our own survival instincts. Perhaps when there’s too much pressure/threat we are more likely to stop thinking about the big picture? Understandable when it’s life or death but surely unnecessary in business, tech or the daily commute?

Technorati Tags: business, strategy, technology

I see many a comment talking about concerns with roundtrips in SOA/distributed architectures. For example in “The Fractal Nature of Web Services” the author starts totalling up additional round trips taken once we disassemble what was a single app into separate remote services. He then goes on to walk down the well worn performance path expressing concern about performance etc.

His concern is certainly valid but it makes a fundamental assumption about how you implement your services:

Each service is represented by some stub that communicates via some protocol to some remote back-end. Clients access the service via the stub and thus incur remote call costs.

Further whether he meant it or not, the author gives the impression that SOA can only be done with Web Services which is absolutely not true. Your choice of method for implementation for SOA is determined by your chosen architectural constraints and can often (and maybe, should always) lead to a hybrid model with “public” interfaces exposed via WS-* or REST and others implemented in Java.

And this is where it gets interesting because with Java we can break the stub/remote back-end constraint. Why? Because we have code-downloading. We can construct a service implementation that offers an interface and is advertised in a service registry as usual but at the point of access, downloads it’s whole self to the client. The service then runs in the client’s address space.

i.e. We have what appears to be a remote service with none of the roundtrip costs because in fact it runs locally.

From a practical perspective that means we can do things like:

1. Create a service that wraps a database and contains all the SQL etc
2. Advertise that service
3. Client downloads service and all it’s dependent code and configuration
4. Client invokes on service
5. SQL connection is made to database
6. Queries are run against database

Now, were the service genuinely remote we would incur two roundtrips:

1. Client to Service
2. Service to Database

But because we can “inject” the service into the client we actually only have the roundtrip to the database. Things to note:

• We keep all the benefits of late binding of services and clients at runtime.
• We taught the client a new protocol on the fly - we downloaded all the client/database communication code at runtime as part of service lookup.

Technorati Tags: distributed systems, Java, soa

Argh, there it is again - the DVLA TV advert that warns of computers constantly trawling for those that haven’t paid their car tax and sending automatic fines. It comes complete with menacing and threatening imagery all in black, white and grey.

Then there’s the railway stations - on my daily journey through Reading and Paddington I’m assaulted with automated brainwash messages about security, late trains, slippery surfaces, surveillance etc.

Has anyone noticed how 1984 this all is? Do we think this is at all a good thing? What does it say about day to day life in our “modern” society? What might it being doing to our creativity, our spirit?

This stuff needs to stop…….

Technorati Tags: personal, philosophy

Normally we class availability as a non-functional requirement of the systems we build but I’m starting to think that’s a mistake because often when we get to project scheduling it’s forgotten. We allocate time, budget and resource to make sure we’re feature complete, that this button or that tick box is visible or that we can gather credit card details but rarely do we treat availability similarly.

Unlike a functional requirement which has some form of physical manifestation availability tends to be invisible dotted around the system in all sorts of different places from architecture and code to hardware. In contrast real features are a logical sequence of inter-linked pieces of code and design that are naturally traceable from entry to exit point. No entry point means the feature is not complete and this can be trivially determined prior to deployment. The first time we know availability isn’t “feature complete” is when our deployed website is down.

Worse, with every single feature or subsystem we add and the corresponding increase in load and size of dependency graph there’s an increased chance that we compromise availability. This can make for some serious growing pains where we add more features, attract more customers and thus more load all the time paying no attention to availability until it’s too late.

Consider the wiring in a house, we add a garage which needs lighting, maybe heating and a motor-powered door. Then we build an extension for guests, more lighting, heating and wall sockets. Eventually, something gives, maybe it’s a piece of wiring that should’ve been replaced or a bad junction but the net effect is to risk burning out all the wiring in the house. Replacing all the wiring would’ve been hard enough without the extensions but now the house is twice as big and the wiring is four times more complex. Worse, we cannot do a piecemeal upgrade instead we must replace all the wiring at once and until it’s done we can’t have all the heating on or we can’t cook at the same time as using the garage.

Like the wiring, availability needs constant and sometimes significant attention and putting the work off can lead to significant cost down the line. Similar arguments apply to other non-functional aspects of our systems such as scalability (it’s no good adding features that attract more customers if the system won’t cope with the additional load).

In light of the above, I believe it might be better to classify availability as a feature if not the feature. After all if the system is not available, none of the cool features we’ve implemented are worth anything because they can’t be accessed. It might also be a catalyst for discussion (especially amongst non-technical staff) of availability’s costs and benefits as compared to those of other features leading to improved strategic thinking.

Relevant Links:

• Vogels on Scalability
• SD Forum 2006 - The eBay architecture
• Inside MySpace.com
• Vogels - E-Commerce at Interplanetary Scale
• A Conversation With Werner Vogels

Technorati Tags: architecture, distributed systems, availability, web

I came across a new concept in my travels this week, an invitation with compulsory acceptance.

Is an invitation that is compulsory to accept an invitation at all? Surely most people would see it as nothing more than a dressed up way of demanding attendance? Surely such “garnish” (Ben Elton fan’s will know exactly what I’m talking about) just engenders bad feeling in many recipients?

Meanwhile in the world of systems…….

We often do the equivalent of compulsory invitations in the way we build software - consider for example the whole dependency injection thing.

In many cases we build some object via a constructor that expects to be injected with a whole heap of other things. How do we guarantee all those things will attend and remain present for the lifetime of our object? After all without these things our object cannot perform it’s intended task. About the only way this works is by insisting that all these things exist within the same JVM as the object to be injected - then they all either exist or don’t exist.

Clearly, this doesn’t work so well in distributed systems where achieving the same availability assurance is considerably more difficult. A slightly more subtle issue is that if our systems are written in the above manner, upgrades are also more difficult because we must manage a dependency tree. e.g. We wish to upgrade System A and it has dependants, Systems B and C both of which will need to be taken off-line whilst we upgrade A.

What is required is a more dynamic form of injection, something that can be reinitialised or changed which at least invalidates injection via constructors (well, unless we throw the whole object out and rebuild it from scratch). In fact, we probably require some kind of event-based solution such that systems can get liveness information about the systems they rely upon and take appropriate action (which will likely include dealing with in flight operations).

Technorati Tags: architecture, design, distributed systems, philosophy

There’s been a lot of stuff going on of late, I certainly feel brighter than I have done in some time. It just takes time to work through things, gain perspective and come to conclusions.

But some things never really get “closed out”. Some things you look back at and wonder if they really were as good as you thought at the time. Were they really worth the effort or did they cost you dearly in subtle ways you didn’t notice? Were they real or simply a function of over-optimism, daring to dream and a little too much rose tinting of shades? Would I have been better off if it had never been?

“What if?”, “why did I bother?”, “was it worth it?”, “why can’t I let it go?”.

Are such things important? Who knows?

Technorati Tags: philosophy, personal

I attended Sun’s London Tech Days conference yesterday. I came away with a couple of overwhelming impressions.

We have serious problems in terms of the quality of our developers. Some of the questions being asked should be reserved for those getting to know the trade at school or university and not be uttered by “paid professionals”. Simply horrifying……and here’s some further food for thought.

Sun have this saying “The Network is The Computer” - based on yesterday I’d say that’s true for all the wrong reasons. See what they really mean is “The Network is One Computer”. There was lots of talk about multiple cores, multiple threads (no mention of the likes of SEDA), race condition detectors (a cool piece of tech by the way), optimizing compilers and so on.

But there wasn’t a single mention of what they would do when you grew beyond a single machine. The closest they came was a few brief mentionings of clustering which only counts for tightly-coupled, uniform computational scenarios. All those web 2.0 sites, Amazon, Google and so on are building systems on top of multiple boxes which need to work together in ad-hoc, changing configurations.

In our current systems construction doctrine we still focus on building our application inside of a single machine out of bits (e.g. Spring or App Server style). Witness how we strive to allow developers to run all that’s required on their own machine and rarely force them to run remotely. Do we really feel this is a good thing given that when the code is given to ops the first thing they do is put it on lots of machines? What incentive does a developer have to write monitoring tools to help ops out when they don’t see the pain in development?

Standard development practices are not aligned with what goes on in deployment, Sun and others need to start bridging that gap and developers need to start working in a style that fits much better with what goes on “over the wall”. Certainly it’s not going to be easy given we have such a long legacy of confining ourselves to a single machine but it’s necessary.

The fact of the matter is we know developers don’t like threads, transactional memory might be a nicer model but it won’t scale any better and virtualization makes it easier to spread software across multiple “machines”. The future looks like it will be multiple, co-operating, separate (possibly remote) processes - now where are the tools etc to make that happen?

Technorati Tags: distributed systems, engineering, sun, technology

Having worked as an independent contractor for a number of years, I’ve finally been tempted to take a permanent job by Betfair. My decision to join them was driven by several factors:

1. The size of the challenge - Betfair’s systems are big, getting bigger and doing serious numbers
2. Ability to influence - as a contractor you are often pulled in to fire-fight or work within what has already been done you don’t get to innovate or change
3. The tech team - I gotta get more of these guys to blog. For now, I’ll point you to Ewan Silver
4. The kind of challenge - it’s broad and not about Jini

Those of you who know my technical background might be able to guess at the sort of things I will be doing. I hope to provide more specifics over time but for now, speculate away!

Some of you may be wondering what this might mean for Apache River and/or Blitz. I think this change will benefit both projects:

1. Blitz will return to being a fun side project with I believe an attendant rise in my enthusiasm - which you can already see in 2.0alpha
2. Apache River could really do with real-world experience such as that which might be gained from working in the Betfair environment

All that remains for me to do is thank all those who have supported me one way or another during my “Lone Crusader” years. That would include Vertex, Tangosol, various at Sun Microsystems (including alumni) and a too-long-to-mention list of individuals - good relationships I plan to maintain.

Technorati Tags: betfair, personal, technology

Ever had the feeling you just weren’t the kind of person someone had in mind when they designed a form?

A couple of weeks back I was filling in such a form which was asking for details of all the companies I’ve worked for over the last 5 or so years. In addition, they wanted contact details for my line managers at those places of work.

Hmmm, you kind of know they want to get references from these places which is difficult when your current company is your own and the previous one was joint-founded by you and others. I just knew as I wrote the details down it wasn’t going to work. Sure enough…..

“Could we please have contact details for business referees that can confirm Mr Creswell’s involvement with Lone Crusader Ltd“.

There now, wouldn’t it just have been eaasier to do what everyone else does and ask for referees rather than last places of work and line management?
Being outside the demographic is sometimes a compliment and sometimes a curse…….

Technorati Tags: humour

How often do we admit something’s wrong and take corrective action? How often do we admit something’s wrong before it gets well out of hand?

There are two basic things we need before we can admit something’s wrong and address the problem:

1. Be strong enough to admit there’s something wrong
2. Be aware that something is wrong

The first item is about having sufficient strength of character to admit our mistakes, learn and move on. Inhibitors for this include ego or pride, fear and external environmental factors (e.g. a violent spouse).

The second item is about recognizing the indicators of a problem. For us humans indicators can be anger, frustration, lack of drive, fatigue, health problems or a constant need to distract oneself via activity. In terms of computer systems, indicators might be Java exceptions under odd circumstances, strange lockups, weird performance issues etc.

The art whether in the human or systems world is detecting these problems early on and reacting to them before they get out of control. In the systems world, so many times the first we know of a problem is when we see an exception trace or the pager beeps at 2am but is this really necessary?

Could we not have been aware of the problem earlier? We put a lot of effort into testing but considerably less effort into endowing our systems with monitoring and feedback mechanisms that can provide us with useful statistics. These statistics are the things that can give us prior warning of a problem and come in many forms:

1. Too many tasks
2. Too many threads
3. Queues that are too long
4. Time per task increasing

Frequently we build logging into our software but this is really a postmortem facility something that is only useful after the problem has occurred. Our OS’en, hardware and software platforms often have monitoring infrastructure built-in, why is it that we so often choose to build our applications without similar facilities?

Technorati Tags: monitoring, performance, systems health

I’m currently working on getting 2.0-alpha3 out the door which means all the usual release procedures and in particular a lot of thumb-twiddling whilst various tests run. People often ask about the testing that goes into a Blitz release and so, given that I’m suffering from test-induced thumb-twiddling it seems like this would be a good time to write something down.

Blitz is basically a layered Jini service with remote wrappers cleanly separated from the core JavaSpace implementation. Thus we can thrash the core without needing to do a full remote deployment which provides many benefits such as eliminating a lot of network I/O that can hide race conditions and deadlocks.

My test machine is a dual processor dual core machine (2 x 2.6Ghz Xeons with HyperThreading, 2Gb, 2×70Gb SCSI Disks) and it’s main duties are to run the long term soak tests for long periods of time (unit tests get run as part of day-to-day development on my PowerMac Dual G5). There are two basic types of soak test that get run regularly they are:

1. TxnStress - a multi-threaded test which fills a Blitz instance with a fixed number of entry’s and then randomly takes and re-writes one of those Entry’s with each take/write pair done under a separate transaction.
2. Stream - a multi-threaded test which writes a sequence of Entry’s into Blitz whilst another thread takes them. Writers and Takers are run in pairs up to a configured number.

Whilst these tests run I typically have jconsole hooked up plotting memory consumption so leaks can be detected and fixed. In addition, Blitz’s in-built statistics API is used to check queue sizes, number of entry’s etc (this is the same API used by the remote dashboard). Finally, each test has code that ensures we are not leaking Entry’s or missing them. Blitz also has a debug-mode where one can simply connect to a pre-configured socket and trigger the dumping of statistics to the console at any time.

These tests get run against Blitz in both transient and persistent mode (which is a mere flip of a configuration variable).

Stream is just about done, so I’ll be shipping a release to SourceForge shortly.

Technorati Tags: javaSpaces, jini, Blitz

Dave has some thoughts on the recent W3C “Web Of Services For Enterprise Computing” get-together.

He notes:

“Given all the angst about Web architecture vs Web services architecture, I was also surprised that there was no support for technical reconciliation. I suggested WADL (Web application description language), to help with enterprises and the desperate perl/python hacker building stronger typed REST services. And for the flipside, I suggested improved SOAP to URI/XML bindings so SOAP/WSDL services would be more easily consumable by REST clients.”

No support for technical reconciliation? Why would either side want technical reconciliation when each believes it’s approach is “right” for it’s own needs?

I think he hits the nail with a comment earlier in the posting:

“There was almost no discussion about how enterprise computing is different from “non-enterprise computing””

Which is sad but not surprising - a good amount of the reason we have the whole Web/WS-* debacle is down to the fact that this kind of discussion didn’t happen when it should have. And if we still aren’t having the discussion, technical reconciliation is likely to be impeded.

Technorati Tags: distributed systems, enterprise, web

Apparently not Anne:

“…..But I’m not so good at blending into the traditional, primarily because I’m just not interested in doing that.

A further issue is that though I have the background to comment on enterprise software, it doesn’t excite me anymore. And every minute I spend looking at enterprise software is a minute I’m not looking at the latest in social media and social tools and new social forms on the web, my real area of interest…..”

For all that enterprise is big business and pays the bills, in many cases it’s not all that interesting. Once you’ve done a couple of integration jobs, you’ve done them all.

All of which leaves me wondering:

1. Might there be some connection between repeated reinventions of the wheel and the run-of-the-mill sameness of most enterprise jobs?
2. How much talent can one really expect to attract into enterprise?

Technorati Tags: enterprise, technology, web

In the last couple of years we’ve seen the arrival of the mashup which is at least on some level nothing more than the latest in a long line of terms for integration. Thus far most mashups consist of a simple amalgamation of a couple of services which leads to a very flat graph of service dependencies. Service dependencies are things like:

1. Data Schemas - structure of data provided by services
2. Endpoints - location of service be it a URL or a WSDL endpoint
3. Availability - whether a service is available for use
4. Reliability - whether a service that is available behaves as expected

These mashups are already at the mercy of the underlying services they are built on. The mashup provider has little control over these services. For the most part these mashups work but it’s because they have only a few moving parts such that the likelihood of issues is low.

Many enterprises have considerably deeper dependency graphs inside their firewalls and have to work hard to keep them stable. There’s probably some limit to what can be achieved once the dependency graph gets beyond a certain depth and it might well be that the maximum depth is smaller once external services are brought into the mix. The maximum depth is likely further reduced because these enterprises wish to treat external services as if they are part of their organization. They want to be able to integrate them using transactions, they want them to have the same level of reliability as what lives within their own data-centres, they want integrated security options etc.

I suspect that a lot of what can be done in a single enterprise (at substantial cost) such as high reliability is going to be considerably more (prohibitively?) complex to achieve across organizations. This is because the level of control required to achieve these targets is beyond that available across enterprises.

Right now I think there’s much effort being made to paper over these issues such as features in WS-*, SLA’s etc. I wonder if it might it be better to give up on this idea of control and build some simpler solutions….

Technorati Tags: distributed systems, enterprise, utility computing, web

I’m not talking personal hygiene which of course is important (your Mum is right, brush your teeth, bathe regularly etc) I’m talking protocols.

Pete’s highlighting the issue which comes down to this strange comment from Paul.

SOAP isn’t RPC? Well sure if I read this I can see the statement Paul references:

“SOAP is fundamentally a stateless, one-way message exchange paradigm, but applications can create more complex interaction patterns (e.g., request/response, request/multiple responses, etc.) by combining such one-way exchanges with features provided by an underlying protocol and/or application-specific information.”

Trouble is the same document also says:

“[SOAP Part2] defines a data model for SOAP, a particular encoding scheme for data types which may be used for conveying remote procedure calls (RPC), as well as one concrete realization of the underlying protocol binding framework defined in [SOAP Part1].”

And then (here):

“One of the design goals of SOAP Version 1.2 is to encapsulate remote procedure call functionality using the extensibility and flexibility of XML. SOAP Part 2 section 4 has defined a uniform representation for RPC invocations and responses carried in SOAP messages.”

What does [SOAP Part2] say?

“One of the design goals of SOAP is to facilitate the exchange of messages that map conveniently to definitions and invocations of method and procedure calls in commonly used programming languages. For that purpose, this section defines a uniform representation of remote procedure call (RPC) requests and responses.”

If we go by the letter of the base-spec SOAP isn’t about RPC but if we look at the spirit, it’s stated that a design goal for SOAP is to support RPC. Note the number of times RPC is mentioned in the base and adjunct specs. Seems like RPC is a fairly fundamental part of this whole thing. Confused?

Update: Don Box wrote up this brief history of SOAP which also makes the link to RPC

Technorati Tags: distributed systems, SOAP, technology, web