For a long time, software development within many an enterprise is treated as a subservient entity. Something that is expected to comply with the demands of the business without complaint and with limited options for pushback.

I believe the software we produce should be viewed as a stakeholder in it’s own right. It has it’s own needs for survival and ongoing growth and if these are always placed behind everyone else’s (i.e. the business) considerations, the results will be a long slow, painful death where the software becomes more and more brittle, less and less maintainable and staff productivity drifts down as staff turnover creeps up.

Consider a car - it has needs, new tyres, oil flushes, new exhausts, paint chip removal, new springs, new clutch etc. Fail to address those needs and your car will turn into a pile of rust before your eyes with all the attendant issues of depreciation, lost investment, breakdowns etc.

Why do we refuse to accept that software has a need for the motoring equivalent of oil changes and the like? Probably because software is an invisible abstract thing such that only those working with it see the damage being done, problem cars are easier to spot for a greater percentage of the population. This isn’t really an excuse however because software gives warning signs just as cars do. If there’s steam coming from under the bonnet (hood) you’d go to a mechanic, if your software keeps crashing or upgrades keep failing or development takes longer and longer it surely follows that it’s time to visit with your software engineer.

[ You may have noticed I like car analogies - here's another: You can't endlessly tune a car, it will only go so fast. Any further increase in speed can only come from starting with a new car that has better basic performance ingredients. The same is true for software, eventually you need a redesign to make further progress because the initial assumptions you made have all been invalidated. ]

Technorati Tags: development, software, systems

Distributed systems practitioners get really excited by swarm theory because it holds the promise of being able to assert a state change in a system without centralized control or a global interaction (e.g. two-phase commit). Bees, ants and the like manage to conduct a co-ordinated life within a co-operative using only local interactions. They don’t have to communicate with the entire colony to agree a good place to nest or a good source of raw materials. It should be no surprise then that distributed systems practitioners also get excited about epidemic behaviours given that they possess similar qualities.

These natural systems contain a good measure of resilience as well. Each bee for example carries a little bit of state with it that it communicates (gossips) with other bees it meets. This state is the result of something it’s encountered in it’s environment. Should this bee die losing the state it’s unlikely to matter as another bee will probably encounter the same environmental conditions and thus the state will be recovered. Victory through weight of numbers.

So nature has provided us a mechanism that can make a binding decision using very loose co-ordination and eventual consistency, whilst functioning entirely off local interactions - no centralized control. Naturally more scalable. What we don’t get is a predictable time for the point at which this decision will be made (concurrent transactional systems aren’t entirely predictable in this respect either). In addition for certain kinds of decision, it’s entirely possible to have race conditions leading to a less than optimal choice. For some systems this doesn’t matter (nature is happy to be a little sloppy) but in cases where it is important there are solutions available.

One key challenge in building these systems is that much of our existing communications middleware is either point-to-point (e.g. RPC or RMI) with fixed addresses or broadcast (message queues or multicast) neither of which is well suited to the random one-to-one nature of gossip driven approaches such as those described above. What’s needed is some form of registry which can cope with dynamically changing membership that takes account of locality of nodes and an efficient means of directed inter-node communication algorithm which mimics the relatively random properties of gossiping.

Technorati Tags: biology, distributed systems, epidemic, swarm, theory

I was reading this from Bill which follows on from Joe. And had a couple of thoughts:

1. Google seems to have applied N > 1 to everything, not just storage - that’s pure distributed thinking, not the norm for the majority of software heads.
2. Eventual consistency might be kinda like concurrent programming for most people - i.e. many like to program sequentially and with the certainty that x has been completed immediately, in order and within a known timespan. Concurrency, eventual consistency and friends aren’t terribly amenable to this programming approach and it consequently melts many a brain.
3. We need for a lot more people to understand CAP.

[Note for Bill should he read this: it seems like your comments are broken, I'm seeing server errors when I hit post]

Check out this job spec.

Notice anything interesting? It’s for a seriously heavyweight distributed systems engineer sure but look deeper. Do you see mention of a single piece of technology? J2EE? JavaScript? Ruby? No, right? How weird is that? How many job specs do you see like that? Surely what matters is whether you know JBoss or Websphere, Java or Erlang or Ruby?

What’s the deal? It’s recognition of the fact that building systems is about how you think and reason which requires sound understanding of theory and how to apply it. It doesn’t matter how much code you can write or in what language because delivering a project is about a whole lot more than code.

So often I see companies create job specs for engineers where the key requirement is to hire someone who can hit the deck coding like mad using whatever tools have been selected. To that end they load the specs up with endless tech hubris and at interview ask the details of this or that bit of syntax or API call. But what about the next project within the company where the tech is different? All those engineers that just got hired are now useless, they don’t have the skills and we lose time whilst they learn. Or we could fire them and hire another lot?

Of course what happens more often than not is that companies ensure they don’t use new tech. Instead they force new projects into using all the same stuff they used before. This is a design disaster as now technology is dictating not design or suitability to requirements. A company that follows the hit the deck coding mantra just has deathmarch and no career progression stamped all over it.

Keeping an eye on trends and keeping abreast of new technology is a good thing to do but the larger context of what to use when, when to build rather than buy, when to dump something because it’s warping the design, when to dump one design approach for another (e.g. going from centralized to distributed) and so on is what really matters. This requires thinking, not an encyclopedic knowledge of a huge number of technologies.

Tech is for sissies - Concepts, principles, patterns, measurement, theory and so on are what matters.

[Confession: The title for this entry was inspired by a recent piece from Pat Helland, one of my favourite thinkers]

Technorati Tags: design, philosophy, engineering, software, systems

All my life
I’ve denied
Ever knowing what its like
You came around
You shook my ground
Now i’m searching for a drug to come down
You’re where I thought I’d never go
I can’t believe I did

Look out below
I’m letting go
Look out below
I’m falling completely
I lost control
I let it go
Now I can see so clearly around me
You’re everything I need

Closure (featured on the soundtrack for Billabong Odyssey - check out those waves).

A little known fact in distributed systems is that once you make them resilient in the face of failure, handling upgrades is relatively simple. Why is this?

To make a distributed system resilient in the face of failure requires that we eliminate all single sources of truth. Truth must be maintained by and sourced from a collective which can maintain all relevant state in the face of nodes joining, leaving and failing.

There’s an additional subtlety to resilience which is that it requires scalability and in particular we need to be able to spread load across the collective whilst the members change otherwise the loss of a node will potentially mean we can no longer handle the current load in our system (we’d no longer have enough processors to cope). Likely as not, to make this work will require us to relax system constraints enough to allow truth to propagate asynchronously.

How does all this help with upgrading though? Because the typical upgrade cycle for a node looks like this:

1. Shut down a node.
2. Upgrade software on node.
3. Restart node.

Such a sequence of events when considered from the point of view of other nodes looks like failure - the node disappears, later returns and needs to be re-knitted into the collective.

Sadly, nothing comes for free so to make upgrading work we need to ensure a level of backward compatibility between versions of the software on our nodes and we also need to account for this in our communication protocols, however there are plenty of examples to help us here such as http and DNS.

Some references:

• Tom Limoncelli of Google mentions this in a talk at Lisa ‘06.
• Bill Clementson in a recent blog about Erlang but it’s largely relevant to any kind of distributed system.

Technorati Tags: availability, design, distributed systems, versioning

Elliotte Rusty Harold with an interesting commentary on the REST vs WS-* war.

There’s one statement that I might contend with just a little:

"The WS-* community really believes that developers are too stupid to be allowed to manage themselves. Developers have to be told what to do and kept from getting their grubby little hands all over the network protocols because they can’t be trusted to make the right choices."

The WS-* community may well see things this way but I think there’s at least one other possibility that would place the fault elsewhere. Given that an awful lot of developers are heard to utter sentences like…

"I don’t want to be bothered with the nitty gritty details of network protocols, threads, persistence etc I just want to write my business logic"

…perhaps it’s natural to expect that various entities will construct technologies like WS-*. Developers are seemingly pushing responsibility elsewhere, placing their fate in others hands and paying the price. Perhaps they should be more careful what they wish for?

When we write programs one of the things we seek to do is encapsulate our data so as to allow us to manage our dependencies and keep our code clean. Most languages OO or otherwise provide mechanisms to support this way of working.

The thing about the average database is that it doesn’t really encourage similar behaviour. It is all too tempting (and easy) to just allow everyone to access everything. Whilst we confine ourselves to a single application using the database, the problem is to some extent contained but often what we actually do is allow multiple applications access to the same database. The exact way in which this is done varies:

1. Sometimes we bundle all our middle tier code together even though it has separate roles and responsibilities and integrate all of it via a single database.
2. Sometimes we have multiple applications each running in a different process.

With each application we put on top of the database the problem gets worse increasing the number of invisible dependencies tying unrelated elements of code together by virtue of accessing a shared schema.

What’s happening is we’re sharing too much intimate knowledge across our system, something we’re all taught to fear. The solution is as always to prevent direct access to this intimate knowledge by interposing layers of abstraction. One way to do this is by requiring access to data to be wrapped up behind an interface. Historically we’ve done this by having a system own the database and expose interfaces that other systems can use to get the data.

Unfortunately there is a well-known issue with this approach which is that the level of granularity is wrong and these additional integration interfaces rapidly balloon into complex beasts. What we need is a a database wrapping entity that has a finer level of granularity than an entire system. Then the integration interfaces will be simpler because there will naturally be a less complex schema underpinning this more limited functionality.

What are we talking about? Services. We end up with a system of lots of discrete services each wrapping up their own data storage.

There are other benefits to this approach:

1. Each service can utilize the most appropriate storage option for it’s contained data whilst having zero impact on other services that might have different needs.
2. Each service is an independent entity that can be managed (monitored, deployed etc) separately.
3. Centralized access patterns are more easily broken down which is useful in cases where we deploy across multiple data-centres.

Who would do such a thing?

Technorati Tags: architecture, database, distributed systems, enterprise

…..Three-tier architecture. Three tier architecture is really a logical partitioning of system functions into presentation, business logic and data-access. Of course, some frameworks have attempted to turn this into a physical reality which is fine but many people believe that such systems are or can be distributed which makes little sense - why?

Because as has been said elsewhere placing code far away from the data source makes little sense. If one is to place code away from the data source it’s going to be for one or more of the following reasons:

1. The computational weight demands separate scaling from the computational load inherent in managing the storage of the data.
2. The computational load in the data-storage layer can be better scaled elsewhere.
3. We can offset the additional latency introduced by network roundtrips.

Most three-tier architectures fail to satisfy any of the above criteria and thus aren’t good candidates for distribution.

…..Synchronous. This is because, in order to offset latency we must exploit asynchronous behaviour. Note that this does not imply the use of messaging rather it means adopting suitable asynchronous design patterns which can be implemented via RPC or messaging.

…..Completely consistent all of the time. Trying to enforce ACID properties across a distributed system is opening an enormous can of worms where one constantly attempts to defy the nature of the network. It’s not impossible to achieve but there is a tradeoff to be made. It’s often better to prefer eventual consistency in good-enough time i.e. something that approximates total consistency under most circumstances whilst degrading (hopefully gracefully) under load or in the presence of failure.

Many have attempted to implement distributed systems whilst falling into one or more of the traps above. Many have paid the price and many have consequently pronounced that distributed is inappropriate, impossible or insane. This is the motoring equivalent of strapping a massive turbo to an unmodified engine and complaining when the pistons explode through the bonnet and the oil is dumped all over the floor. In both worlds the remedy is the same, talk to an expert and be prepared to throw out a few beliefs.

Technorati Tags: architecture, distributed systems

Tom Ayerst pretty much hits the nail on the head.

I would suggest just one more refinement:

Architecture and code need ongoing concern, review and re-organization. One simply cannot leave what has been previously built untended and focus on the next feature. It only takes one broken window……

My notes on the talk by Werner Vogels and Swami Sivasubramanian:

State management is the dominant factor in scaling - this is the stuff that is tough to look after, stateless is easy.

There’s a tight, complex interplay between scalability, availability, consistency, efficiency, management and performance.

Consider that billions of your body’s cells commit suicide in a day and yet you continue to function uninhibited. This process (Apoptosis) is essential for the health and stability of the overall organism and can be usefully applied in distributed systems. There are other interesting aspects of our biology that are relevant - check out the paper "The Limits of the Alpha Male"

Amazon is a collection of seven web-sites, it started as a website and a database but is now a distributed system. These changes were driven by the natural brittleness of integration via the database, performance and scaling issues. It was noted that database technology is many years old (reference was made to this article in ACM Queue) and we really need to move on.

For Amazon, incremental scalability is key and it’s desirable to be able to scale dynamically both up and down with demand. Improved performance can be defined in many ways including serving more units or serving larger units such as is required when datasets grow.

An always-on service is said to be scalable if adding resources to facilitate redundancy does not result in a loss of performance. Other aspects of a scalable service are that it:

• handles heterogeneity
• is operationally efficient
• is resilient
• becomes more cost effective when it grows

We should never expect systems to be stable:

• things leave, join and fail continuously
• perturbations and disruptions happen
• failures are highly correlated and systems do not fail by stopping

A key part of Amazon’s approach to defining service contracts is SLA’s. Conventional wisdom for SLA’s is that they are a one-way contract but in fact they should be considered as two-way contracts (what the service promises and how it is to be used). The contract might well include factors around:

• latency in respect of single service or paths through the system
• durability and availability
• cost

SLA’s introduce the right for a service to throttle in the face of various conditions and should not be defined with single numbers, rather they should be defined with ranges.

The remainder of the talk was concerned with Dynamo which has been previously known as HASS due to constraints in respect of an upcoming unreleased paper (titled "Dynamo: Amazon’s Highly Available Key-Value Store" to be presented at SOSP 2007 and my notes say it will be released on August 9th). Dynamo embodies much of what was talked about above, achieving it’s functional and non-functional targets with a mixture of:

• Sloppy quorum and hinted handoff (Werner’s own terms)
• Vector clocks for versioning and consistency, and exposed to the client application which is expected to define the model for merges etc)
• Consistent hashing and other p2p techniques for scalability (I’d recommend examination of examples such as Chord or Bamboo)
• Anti-entropy using Merkle Trees

Update: The paper is now available

Technorati Tags: distributed systems, google, conference, scalability, amazon, dynamo

This scenario get’s played out all the time in IT. The young guns claim that the old guys are out of it, don’t get the latest tech, aren’t smart enough whilst the old dogs smile and are heard to say they’ve seen it all before.

There’s a fundamental tradeoff at work here:

1. Intelligence allows us to at least potentially progress faster
2. Experience allows us to avoid making mistakes as we make progress

Thus a bright inexperienced person may make fast progress but they’re much more likely to make mistakes which will slow them down. In contrast the experienced person may make slower progress but fewer mistakes. Classic hare and tortoise. Who wins?

The nature of software is such that the mistakes we make can take a long time to manifest and when they do, they cost us big time. Thus:

1. Mistakes don’t result in short-term localized damage rather they are far more disruptive with long-term, difficult to clean up damage
2. The time between the root cause of the problem and it’s costly manifestation is large.

It follows that for our intelligence to count we must be able to see sufficiently far ahead to spot our mistakes coming before they get out of hand. Is this achievable? I think software history says it’s not and thus experience is our only tool for understanding root causes and spotting the early signs of an approaching asteroid.

I reckon there’s a lot to be said for the old tradition of master craftsmen handing down their knowledge and experience to apprentices…. (and perhaps the old dogs can learn a few new tricks along the way).

Technorati Tags: process, software, technology

The debate about all these many core processors continues to circle the blogosphere. Tim Bray had this to say which set me thinking (always a bad thing):

Any time we have a piece of state that needs to be accessed concurrently we hit problems. One can hide this problem using messaging (or similar) but the key aspect in these solutions is that we can partition operations into streams against discrete elements of data (a discrete element could be a group of things) that don’t interfere with each other. Partitioning however can be problematic:

1. Our data has to be amenable to partitioning via hashing or some other method.
2. It gets tricky when we need to deal with availability and disaster recovery.
3. Getting the correct granularity of partitioning can be challenging.

Which is interesting because whilst we’ve eliminated the concurrency issue, we’re now faced with a different one (partitioning) which could be just as hard to cope with and requires just as much thought from a developer and/or architect. Coincidentally, Werner Vogels (Amazon) is going to be talking about an internal data store (HASS) at the Google Scability Conference and specifically the problems of partitioning and consistent hashing (my original interest with respect to this talk was in the context of the CAP conjecture).

Another means of avoiding all these concurrency issues is to push them somewhere else. More often than not this becomes an exercise in creating a supposedly stateless system which in reality simply puts all the state in one place, usually the database. The argument is that this is acceptable because it’s only the likes of databases that should deal with these hard issues.

The rub with having the database handle it is that the concurrency model it uses will only scale across so many processors (more if you’re read mostly, less if your not) and cope with so many concurrent accesses from the stateless component. Once again to get our database layer to scale, we’ll need to partition our data into shards across multiple databases (an approach adopted by a number of top-line websites) or find some other way to reduce concurrent load on the database instance.

The act of partitioning can mean we reach a point where we can no longer expect to have atomic updates because the mechanisms for achieving it (e.g. two-phase commit) stop us scaling. When this happens we must construct complex or at least exotic solutions such as that proposed by Pat Helland.

Okay we got rid of our concurrency problem and swapped it for a partitioning problem which then turned into something of an exotic problem. Are we any better off? It seems no matter which way we go we end up with some tough problems to solve.

Perhaps there’s a sweet-spot tradeoff where the combination of a CMT box, with data partitioned across a number of processes and each process containing a simple concurrency model covers most situations. Even if that’s the case it seems developers will have to learn a few new tricks.

Technorati Tags: architecture, concurrency, software, systems

Update: A good comment over on Reddit.

Woohoo, after jumping through some hoops I’m going to be able to attend the conference. I will of course be representing Betfair.

I’d be happy to attend any of the talks but my personal favourite is the storage talk by Swami Sivasubramanian and Werner Vogels. I’m really looking forward to making some new contacts. If anyone can recommend some good Seattle beers to try, drop me a line.

Patrick on the lack of decent reasoning in WS-* world:

“It is precisely this lack of rigor that has failed ws-* from the start”

If this problem were confined to WS-* it’d be a cause for celebration…..sadly it’s pretty much everywhere in software these days.

Update: Was wading through my bloglines feeds and forgot about something Bill said a while back which is relevant.