Pragmatic Dictator

Jim Waldo writes:

My own conclusion is that system design is really a matter of technique, a way of thinking rather than a subject that can be taught in a particular course. It might be possible to build a program that teaches system design by putting students through a series of courses that hone their system design skills as they move through the subject matter of the courses. Such a series of courses would, in effect, be a formalized version of the apprenticeship that is now the way people acquire their system design technique…..

…..Even worse than not being visible to the customer, work done on designing the system is not visible to the management of the company that is developing the system. Even though managers will pay lip service to the teaching of The Mythical Man Month, there is still the worry that engineers who aren’t producing code are not doing anything useful. While there are few companies that explicitly measure productivity in lines-of-code per week, there is still pressure to produce something that can be seen. The notion that design can take weeks or months and that during that time little or no code will be written is hard to sell to managers. Harder still is selling the notion that any code that does get written will be thrown away, which often appears to be regression rather than progress.

In such an environment lip service often extends to technical strategy as well.

From On Designing and Deploying Internet-Scale Services (James Hamilton - Windows Live Services Platform):

We have long believed that 80% of operations issues originate in design and development, so this section on overall service design is the largest and most important. When systems fail, there is a natural tendency to look first to operations since that is where the problem actually took place. Most operations issues, however, either have their genesis in design and development or are best solved there. Throughout the sections that follow, a consensus emerges that firm separation of development, test, and operations isn’t the most effective approach in the services world. The trend we’ve seen when looking across many services is that low-cost administration correlates highly with how closely the development, test, and operations teams work together.

Yep, I’m a firm believer too….

I commented in my previous post on the fact that working with message brokers can lead to tensions where we’re forced into letting the broker do more than we’d like.

I suspect that the bigger the list of features provided in a single broker the greater the probability we’ll be forced to comply with (and attempt to work around) undesirable models and behaviours that aren’t core to our requirements. Thus under certain circumstances a specific solution that addresses one core problem whilst asserting a minimum of additional constraints can be more attractive than a jack-of-all-trades solution.

It also occurred to me that brokers might not be the only piece of software to exhibit this characteristic. RPC systems tend to suffer similar problems having a tendency to tightly bind endpoint addressing, transport and argument marshalling. I also wonder if these issues contribute to some of the backlash against the behemoth that is an RDBMS implementation. For example whilst we might like our RDBMS to look after our data, we might not wish to put up with the querying, concurrency, deployment and management models it also asserts even with a myriad of configuration options.

Technorati Tags: architecture, design, technology

Say the word messaging to a subset of developers and for some reason the immediate knee-jerk is to assume that means using some kind of message broker (Tibco, ActiveMQ or whatever). Utter the term “asynchronous communication” and that is typically equated to messaging and thus also implies use of a message broker.

I find this strange because messaging is possible via a myriad of methods including carrier pigeon, pony express, tcp, multicast, http and many other transports. As for supporting asynchronous operations well that’s governed by the API(s) provided by the transport. In fact this is only partially true because a lower-level synchronous API can be wrapped in an additional layer to produce an asynchronous API. Transports and layering are often seen together, for example it allows us to introduce support for reliable delivery if the underlying transport is not robust (often it isn’t).

There’s no denying that all (or most) of these features (asynchronous APIs, messaging, reliable delivery) are provided in various of the message broker implementations but rarely in the form of a set of composable layers. The preferred approach is usually a myriad of configuration options leaving us at the mercy of the vendor to provide and support just the right combination of configuration possibilities to match our design challenge.

The indivisible nature of these brokers hampers us in other ways too. It can be difficult to use the broker purely for messaging without being forced to work with say its models for routing control and security. I’m sure this is appealing to the vendors but it doesn’t seem like such a great deal to me.

Technorati Tags: architecture, design, messaging

Bill ponders:

“Sometimes I wonder how a deployed n-tiered scale up monolith can be gradually refactored to a scale out model or run on scale out infrastructure. It’s well documented that companies like Amazon and Ebay have done just that, only how they did it tends to get left out of the slide decks. I suspect it involves thinking quite differently about what ‘good’ application code is….”

I think one reason for the absence of this information in slide decks is that it’s a competitive advantage, a barrier to entry. Mastering scale-out allows one company to handle more load and more customers whilst offering more function than another. Why teach other companies how to compete? For me though the real issue is that the transition is a major effort that defies easy explanation via slideware.

Makeup of a Monolith

The rough form of a monolithic system is a logical (maybe physical) n-tier with simple scale out in the places where it’s easy e.g. a stateless web tier. The remainder is typically a single database containing all data which is fronted by some kind of cache (some systems will also talk to legacy systems via various mechanisms and some will have an extra database or two). Some may view their database replication strategy/cluster to be horizontal scaling but to be the real deal would require explicit partitioning or sharding. All code has access to all of the data in the database and it’s typically not well controlled which leads to what I term the “integration via database” anti-pattern.

In essence to move to a scale out architecture requires us to partition up all the state we have held in the database (admitting there’s incentive to break up the middle tier to avoid it getting overly large and unwieldy). Sounds easy but of course is anything but because there’s just so much beyond “thinking quite differently about what ‘good’ application code is” to learn and change. One might expect that we build things up in horizontal slices, create some new infrastructure, port to the new infrastructure, the usual “upgrade the middleware” type approach but this doesn’t work due to lack of hindsight and the sheer size of the challenge. The antidote to this is to take one little step at a time.

Basic Approach

Instead of the “big scary re-architecture” we take vertical slices through the monolith, separating them out so that we can evolve each with a limited amount to consider. We push these pieces all the way through into deployment so as to expose all our processes and tools to this new type of entity. This is not easy but:

1. At least it’s being done against a small vertical slice which makes it a more manageable problem.
2. The transition cannot be managed in one big step, it will be necessary to manage old and new alongside each other so we need to get used to that.

Another option for learning how to work outside of the monolith is to implement some new set of features in this new style and drive that through the delivery chain. Obviously whilst this develops knowledge it’s not breaking down the existing monolith, so let’s move on to consider the basic steps (order is not fixed) for extracting a vertical element:

1. Identify some reasonable sized function or chunk of data to partition.
2. Define an interface which will wrap this function and/or data.
3. Move code and data behind the interface.
4. Modify other code to access code and data via the interface rather than e.g. direct access to the database.

Step (1) often requires us to examine both data and function. Ideally the interface we introduce in step (2) would look like a remote service. It may for some period of time still be part of the monolith and therefore local. In some cases we can’t make the interface remote initially because the refactor would be too complex so we must later come back and recast the interface as a remote service. What form does this new interface take? Ignoring religious issues it could be WS-*, REST, some other form of remote invocation (sometimes custom but not fine-grained method calls) or messaging.

The arrival of this new interface allows us to introduce behaviours such as asynchronous operation and eventual consistency into a controlled area of our codebase. It’s possible that the data we’ve wrapped behind the interface is still being drawn from the original database so we can also consider moving this data to it’s own separate storage mechanism (which may or may not be another database) and introduce sharding and partitioning.

Challenges

Some developers will fall back on “remoteness dogma” stating that remoteness is a performance inhibitor. Indeed they are right but scale is at least (if not more) important and becomes a key focus when we can’t buy a bigger box to make the monolith perform. For me the real issue at hand is the corruption of the mind that comes from thinking transactionally. In this world we focus on minimising the amount of time a transaction takes for fear of lock contention and blocking threads too long. We become obsessed with consistency, feeling compelled to do all work ahead of the actual transaction. This thinking naturally leads to optimising the execution paths heavily and eschewing remoteness. Key to tackling these issues is socialising use of asynchronous techniques and the fact that consistency is in the eye of the beholder (and determined by how often they can observe the relevant data).

During the recasting of a vertical slice into a standalone element it’s usually necessary to ensure it can cope with existing load (and a bit more). Establishing what the current load looks like can be challenging as monitoring and statistics are often centred around information available from the database, application servers, load-balancers and operating systems i.e. the infrastructure. This data is certainly useful but says little about what’s actually happening in the application code. A good way to address this problem and improve the lot of the operational teams is to introduce a programme of application instrumentation that will deliver appropriate statistics and high-level diagnostics.

Inevitably we will be adopting and/or building some new infrastructure but we must be careful how much we try to acquire and/or custom build in advance of real-world experience (see the hindsight link above). Fortunately there’s little that’s actually required up front other than some mechanisms for service location and statistics gathering so the impact of mistakes is usefully limited. Follow-on candidates might include messaging, deployment and security. Remember also that many vendors are still producing infrastructure suited to big-iron monolithic development and single data-centre environments (which can make resilience in the face of certain kinds of failure difficult).

Static configuration (per-machine or in tools) can be extremely troublesome containing a lot of URL’s, server addresses, machine names and database references. All of these items need changing at each stage from development, through testing, QA, staging and production. The move toward a more distributed approach will only make this worse as it creates a need to copy and tweak more configuration on more machines. It’s important from the early stages to focus on eliminating as much of this as possible. In an ideal world a machine would be configured with at most its designated duty and environment (testing, development, production) obtaining everything else it needs from services in the environment.

Wrap Up

To go horizontal is extremely challenging and cannot be addressed by the typical re-architecting initiatives many companies indulge in. There’s too much to learn and change such that the only option is a slow, step by step, learn as you go transition that gradually chips away at the monolith. For Amazon it seems this transition has taken at least five years and over at eBay it looks like they started the transition sometime around 1999/2000.

Technorati Tags: architecture, distributed systems

Alright, we’ve previously established that at least some enterprises have a substantial software investment outside of the classic business process arena. We’ve also seen an example of advice that fails to take account of this class of enterprise. Now it’s time to talk architecture.

The more traditional enterprises (those that only need software for business process automation and support) made a mistake in their past which needs undoing. They focused on building applications not systems. Each application was designed to tackle some individual aspect of their business processes which when they needed to be integrated caused much pain. The result has been a trend (SOA) to break up all those application silo’s into a collection of shared services on top of which appropriate applications can be built by the enterprises themselves or others. In the latter case shared services must in some way be exposed to others outside the firewall.

Creating services as above is merely one method of partitioning code and data. This method does not apply so well outside of business processes so we must find another model. ROA presents one such model, one that is data centric and works well in the world of the Web where we desire ad-hoc (chaotic) assembly of resources (mashups). The nature of the Web is such that it can be difficult to know who your users are (there are too many) and to manage transitions from one version of an API to another. Basically you don’t know what your dependencies are and it can be difficult therefore to measure the impact of changes. Some have suggested that it’s not appropriate to retire old APIs or resources but this can have significant impact in terms of maintenance such that at least some organisations do deprecate old APIs and retire them eventually. ROA like any other architecture has it’s limitations.

ROA is of course derived from REST. REST includes a set of constraints which are essentially just useful architectural patterns. Some would claim they deliver scalability but I prefer to state that they are “scalability enabling”, they don’t inhibit scaling. However it’s important to realise that behind the Web layer one must still build scalable infrastructure. Building this infrastructure the right way (architecturally) yields scalability not REST itself. Many websites rely on running multiple copies of their web application, scaling via their database and caching solution which will often be more than enough.

However there is a class of enterprise website for which this approach fails, because the consistency models provided by databases and the like actually can’t scale as far as is required. A further complication is that managing everything as a single web application becomes impossible. Each part of the application has its own unique demands in respect of tuning, configuring, monitoring and maintenance:

• Tuning for one part of the application has adverse effects on other parts.
• Configuration becomes a nightmare because there are so many different settings to worry about. Something that works well in testing isn’t appropriate for production leading to separate profiles that must be maintained and kept in sync leading to forgotten changes etc.
• Monitoring produces so much of a mixture of data that it becomes a major exercise to filter out just what you need.
• Maintenance becomes an exercise in chasing down long chains of dependencies to make a simple change.

It becomes necessary to break up the application, storage, caching and so on into more manageable pieces that can run separately as a distributed system. Each element provides a service but not as would generally fit with the “classic” definition of SOA. Our requirements for partitioning are driven by multiple forces and thus the decision as to how exactly to break up the application must be determined on a case by case basis. Such a decision could be driven by amongst other things business model (web applications are still surrounded by business processes), scaling needs, specific storage requirements of underlying data or provision of a specific feature for the website.

One might choose to expose such a service using WS-*, messaging, a Web/Resource approach, CORBA or even some form of custom service invocation layer. Perhaps surprisingly there’s a growing number of examples where the custom service invocation layer option is used. I believe this is because all other approaches represent a compromise achieved through limitation of architectural options in such a manner as to be inappropriate for these demanding cases.

We cannot call this architectural approach SOA, ROA, EDA or anything else, it is simply about creating isolated, independent elements and minimising dependencies. It is something we’ve been doing inside of our programs for years. It also allows us to construct a working, manageable system at large scale. It is common sense. CSA anyone?

Technorati Tags: architecture, distributed systems, scalability, software

Check out this article from Computing. It is apparent good advice for SOA implementation but as mentioned in my previous post, something has been forgotten - some enterprises provide software as a web app that is their product and revenue generator. This software could be rendered into services behind the firewall, yet is not about business processes and must be treated differently.

A quote from the article:

Mistake No. 3: Leaving SOA to the techies

When the SOA process is left mostly with the IT side of the organisation, services risk being designed to optimise software performance and reliability, but may not fully reflect the business requirements.

Clarity of business interfaces is essential for cross-application integration or multi-organisation use.

What about an interface that provides a specific website feature and is a service in it’s own right? Such an interface is unlikely to be exposed across organizations because it provides a business specific feature we do not wish to share with others. Further such an interface probably has few business requirements though the underlying service may need to support auditing or customer tracking tools.

A further quote:

Mistake No. 1: Irrational SOA exuberance

Excessive numbers of services ­ those that cannot be readily matched to the business model of the application ­ are a sign of an SOA environment where applications need to be checked as they are completed.

Such environments may feature repositories full of services, volumes of documentation and an impressive collection of new tools and middleware, but what they will not have is agility, incremental software versioning or reuse.

Again let’s consider a service that provides a website feature such as recommendations. How much does it have to match the business model? One might argue that SOA is only concerned with business processes but surely we can model other things as services?

So what exactly is a service, what is SOA and where does REST fit in? I’ll cover that next….

Technorati Tags: architecture, distributed systems, soa

One designs or architects solutions to problems - technologies be they ESB’s, CORBA etc are merely implementations of (parts of) an architectural solution. Note that one might compromise one’s architecture slightly because it’s expedient to use an off-the-shelf product. Regardless the important point is that architecture comes first, not last and yet that’s rarely the case these days. Instead technology rules…..

Consider how prevalent the use of frameworks is within our industry and think about the fact that in many cases one simply writes a POJO or two and leaves the rest to the framework. The framework makes life easier, it solves the big problems but it also exerts force on the design of our software as after all we must write it to follow the appropriate conventions, implement the appropriate methods etc.

The very worst example of the framework trend is seen in the decision to purchase a mammoth framework offering that provides everything in one box as an “integrated solution”. A huge stack that get’s connected into everything and exerts massive gravity on our architecture. Everything becomes an exercise in warping aspects of our system to fit with this stack and the assumptions of its creators. Essentially we’ve bought “architecture in a box”.

We’ve been doing this sort of thing for a long time and there’s even a business case to go with it. Enterprises want commodity developers. These chaps are not trusted to take on the bigger challenges rather it’s deemed appropriate to use frameworks (in the form of middleware) to address these big issues and confine the developers to the task of simply implementing the business logic.

There’s an entire industry of analysts and others devoted to producing endless tech comparisons to determine just which of the myriad of frameworks will be the single, final silver bullet solution that allows one to implement an entire system in a matter of weeks. This stuff is gobbled up by the the commodity programmer brigade.

Changing these behaviours is challenging: Architecture RIP? Possibly but a recent comment on ESB’s from Ron Schmelzer means there’s still hope. And there is at least one part of the developer universe that sees value in architecture.

Technorati Tags: architecture, design, technology

The longer one holds onto the single shared memory, multi-core, big box approach, the harder and more costly it gets to shift to distributed.

Every time we buy a bigger box for increased load we’re wasting money come the day there isn’t a bigger box to buy (something that is looking increasingly likely for many of us). All that money would have been better spent on buying racks of smaller boxes. It’s possible we can recover some of our losses by repurposing that big iron via virtualization rather than throwing it away (like all our previous big boxes) but of course, if that box dies it takes an awful lot of VM’s with it.

Every time we assume we can keep all our data in a single memory or database (even if it’s a cluster) we’re embedding assumptions into our software that will be broken come the day we must partition across multiple memories or databases.

Each time we choose an algorithm that doesn’t easily partition or assumes a single memory/database we’re storing up trouble in our data and computational models.

In big monolithic systems it’s possible to create (by force) a never-fails environment which allows developers to ignore various edge cases. The move to a system built out of many separate parts makes failure almost impossible to avoid. This requires us to adjust our system design to take account of all those edge cases we previously ignored.

The time we spend gaining experience in building big monolithic systems has limited application when we switch to building distributed systems. We must learn new habits and adopt new modes of thought and that costs time.

In the worst cases, an organization’s processes, tools and departmental structure become heavily optimized for managing these big monolithic software and hardware systems such that it needs serious revision to cope with the move to horizontal, many box scaling. Typical problem areas include:

1. Monitoring - suddenly there’s a much greater number of machines to gather stats from. Existing gui representations mightn’t cope with such a large number.
2. Diagnosis - no longer does a single timestamp imply an order on events making analysis of logging information and root cause identification harder.
3. Deployment - previous methods simply break as the level of automation provided is inadequate for the number of machines and software components involved.
4. Testing - existing testing practices where everything can live on the developer’s desktop or in a single VM are no longer viable. There are too many moving parts and the convenience of isolation provided by testing at the desktop or in a single VM is lost.

I doubt threads will ever go away but learning to build and manage systems constructed in any of the following ways might be worthwhile:

1. Multiple communicating reliable processes on a reliable bus
2. Multiple communicating unreliable processes on a reliable bus
3. Multiple communicating unreliable processes on an unreliable bus

[ Where bus is typically a backplane or a network ]

Technorati Tags: architecture, distributed systems, process, scalability

There’s been some renewed discussion on the relative merits of push and pull for circulating changes.

What I find fascinating is how there’s often a tendency to polarize solutions one way or the other - either we’re entirely push (with failover support etc because we absolutely cannot afford for it to fail) or we must be entirely pull (and worry about what speed of polling to use and build infrastructure that can scale with it etc).

The Good and Bad of Push

Push allows for timely delivery of information updates. If the rate is high enough it makes sense to batch updates together for more efficient delivery. Significantly from the perspective of most, push ensures that we burn CPU cycles as and when there’s something worth doing in contrast to pull where we can waste cycles (though some can be saved with e.g. appropriate use of caching) finding out nothing has changed.

The downside to push comes when clients can’t receive their updates due to network partition or their own downtime (failure, running out of battery power, whatever). When this happens, if we stay push focused we must build appropriate mechanisms for tracking what messages a client has or has not received and hold on to them which can get messy/complex.

And how do we know the client is back? Because it will reconnect, it will pull if you will…..

The Need to Pull

Pull allows a client to dictate when it receives it’s updates and can be particularly attractive in the case of slow update rates. Pull also allows us to recover from various lost event scenarios like:

1. Delivery failure - given a rough idea of rate of event delivery and a period of silence (that is no event has been received) we can perform a check for lost events by performing a pull. And a failed pull tells us quite clearly something is broken.
2. Client offline or dead for some period of time.

Recovery is performed by going back to the "event archive" and finding all the events we missed (we can easily do this so long as we have noted the last event we’ve seen, this works really nicely if we do batching of events) after which we can return to the push mode of operation.

We can limit the size of the archive somewhat by bounding the maximum amount of time a client can be down for whilst still being able to restore itself.

To make this work requires that we provide some way to identify each event uniquely and the ability to page through the "event archive" efficiently.

The Best Of Both

Rather than focus solely on either approach in isolation, I think the best solution is to use a combination. This has a couple of advantages:

1. Clients can potentially use whichever method is more appropriate for them.
2. It provides significant opportunity for fine tuning.
3. It provides a nice simple recovery model.
4. Responsibility is balanced throughout the system keeping complexity down.

[ I’m not alone in this belief as Bill describes exactly such a hybrid approach from the perspective of his favourite technologies (I quite like them too). What I wanted to do was describe the underpinning patterns because I believe this allows us to be technology agnostic and build a working system in whatever environment we’re faced with (for example JavaSpace05 could be used as a substrate). ]

Update: A variation on the scheme allows a client to pull some base state and a set of events from the archive after which it resumes listening to events. The size of the archive can then be managed by every so often updating the base state and storing events since then - basically we’re checkpointing.

Technorati Tags: architecture, distributed systems, events, messaging

I was particularly interested in this part of a recent post from Coté:

James knew that there was a vast schism here between the simple-heads and the complex-heads. The REST and the SOA heads. The WS-Deathstar vs. the WS-* people. Essentially, the two camps don’t want to have anything to do with the other because each thinks the other is fatally flawed:

I think the root cause of the conflict comes down to differences in underlying design philosophies…..

Challenges

In building their systems both sides must tackle challenges such as:

• Failure
• Scaling
• Data Partitioning
• Persistence
• State Management
• Security

Two Different Approaches

Most of enterprise buys into the middleware concept where all the challenges are to be handled inside of an all-encompassing, tightly integrated framework leaving the “trivial” matter of writing some custom logic to satisfy their specific requirements. Cramming all these things into a single piece of infrastructure inevitably leads to complexity in many forms including large surface area (see the section titled “Simplicity as its Own Virtue”), big hardware, big configuration files and so on.

In contrast stands the web crowd that prefers de-centralized, distributed, layered solutions where all participants share responsibility in handling these challenges and application developers are expected to design accordingly (say by implementing retry strategies, idempotency and the like rather than transactions).

Incompatible Views

Essentially the enterprise crowd tackle these tough challenges with technology whilst the web crowd and others tackle them with design. One side complains of toy technologies because they expect the difficult stuff to be tackled in their underpinning middleware monolith and the other complains of complexity because they prefer to emphasise design and simple components.

Bullets

SOA isn’t (or at least shouldn’t be) about technology. The systems aspects of SOA can be constructed using many different technologies (even a mixture) but many still think SOA requires WS-Deathstar. Similar issues exist around REST which many mistakenly associate purely with http. This speaks to a much bigger ill that is rife in our industry - the tendency to conflate concept and technology.

It can surely be no surprise that the middleware mob find distributed hard, given that all they know and all that they do is pretty much in complete opposition to what is required for such systems.

An oft-cited reason for the existence of complex middleware is enterprise’s desire to utilise commodity programmers. Little wonder that so called rock star programmers have zero interest in such a world and are to be found playing around with the web.

Technorati Tags: architecture, design, systems

When we write programs one of the things we seek to do is encapsulate our data so as to allow us to manage our dependencies and keep our code clean. Most languages OO or otherwise provide mechanisms to support this way of working.

The thing about the average database is that it doesn’t really encourage similar behaviour. It is all too tempting (and easy) to just allow everyone to access everything. Whilst we confine ourselves to a single application using the database, the problem is to some extent contained but often what we actually do is allow multiple applications access to the same database. The exact way in which this is done varies:

1. Sometimes we bundle all our middle tier code together even though it has separate roles and responsibilities and integrate all of it via a single database.
2. Sometimes we have multiple applications each running in a different process.

With each application we put on top of the database the problem gets worse increasing the number of invisible dependencies tying unrelated elements of code together by virtue of accessing a shared schema.

What’s happening is we’re sharing too much intimate knowledge across our system, something we’re all taught to fear. The solution is as always to prevent direct access to this intimate knowledge by interposing layers of abstraction. One way to do this is by requiring access to data to be wrapped up behind an interface. Historically we’ve done this by having a system own the database and expose interfaces that other systems can use to get the data.

Unfortunately there is a well-known issue with this approach which is that the level of granularity is wrong and these additional integration interfaces rapidly balloon into complex beasts. What we need is a a database wrapping entity that has a finer level of granularity than an entire system. Then the integration interfaces will be simpler because there will naturally be a less complex schema underpinning this more limited functionality.

What are we talking about? Services. We end up with a system of lots of discrete services each wrapping up their own data storage.

There are other benefits to this approach:

1. Each service can utilize the most appropriate storage option for it’s contained data whilst having zero impact on other services that might have different needs.
2. Each service is an independent entity that can be managed (monitored, deployed etc) separately.
3. Centralized access patterns are more easily broken down which is useful in cases where we deploy across multiple data-centres.

Who would do such a thing?

Technorati Tags: architecture, database, distributed systems, enterprise

…..Three-tier architecture. Three tier architecture is really a logical partitioning of system functions into presentation, business logic and data-access. Of course, some frameworks have attempted to turn this into a physical reality which is fine but many people believe that such systems are or can be distributed which makes little sense - why?

Because as has been said elsewhere placing code far away from the data source makes little sense. If one is to place code away from the data source it’s going to be for one or more of the following reasons:

1. The computational weight demands separate scaling from the computational load inherent in managing the storage of the data.
2. The computational load in the data-storage layer can be better scaled elsewhere.
3. We can offset the additional latency introduced by network roundtrips.

Most three-tier architectures fail to satisfy any of the above criteria and thus aren’t good candidates for distribution.

…..Synchronous. This is because, in order to offset latency we must exploit asynchronous behaviour. Note that this does not imply the use of messaging rather it means adopting suitable asynchronous design patterns which can be implemented via RPC or messaging.

…..Completely consistent all of the time. Trying to enforce ACID properties across a distributed system is opening an enormous can of worms where one constantly attempts to defy the nature of the network. It’s not impossible to achieve but there is a tradeoff to be made. It’s often better to prefer eventual consistency in good-enough time i.e. something that approximates total consistency under most circumstances whilst degrading (hopefully gracefully) under load or in the presence of failure.

Many have attempted to implement distributed systems whilst falling into one or more of the traps above. Many have paid the price and many have consequently pronounced that distributed is inappropriate, impossible or insane. This is the motoring equivalent of strapping a massive turbo to an unmodified engine and complaining when the pistons explode through the bonnet and the oil is dumped all over the floor. In both worlds the remedy is the same, talk to an expert and be prepared to throw out a few beliefs.

Technorati Tags: architecture, distributed systems

Tom Ayerst pretty much hits the nail on the head.

I would suggest just one more refinement:

Architecture and code need ongoing concern, review and re-organization. One simply cannot leave what has been previously built untended and focus on the next feature. It only takes one broken window……

Intel wades in on the “we can’t do any more magic concurrency for software” issue.

It’s been debated often enough and always seems to come down to the fact that the average programmer isn’t able to cope with concurrency and needs higher levels of abstraction to do it for them. The thing is, we already have such abstractions e.g. transactions and we know they can only take us so far. Worse there are other abstractions out there such as blackboard systems which these average programmers either can’t or won’t try to cope with.

So what is to be done? Well if the last couple of decades are anything to go by, absolutely nothing! Why? It’s the talent limit. How many chip designers are there in the world? How many motherboard designers? How many car designers? How many developers? I’m willing to bet that there are considerably more people in the developer category than any of the others. This is because we’ve lowered the bar in terms of developer quality to cope with the wide demand for bums on seats and lets face it, it’s unlikely that the average enterprise is going to change it’s policies in this respect. It’s interesting to note that it’s much harder to lower the bar in for example chip design, it either works or it doesn’t whilst software is almost expected to be flaky these days.

Until we decide to clean house in software land, we’ll not get progress on these thorny issues because they only matter to the few and when all said and done, there is a school of thought that might suggest that software is good enough, concurrency, efficiency, quality and green’ness be damned.

Update: An example of how challenging it can be to make concurrency easy can be found here. On the surface we’ve done some good things and yet we are still open to the simplest of errors (in this case a missing synchronized clause).