Pragmatic Dictator

The longer one holds onto the single shared memory, multi-core, big box approach, the harder and more costly it gets to shift to distributed.

Every time we buy a bigger box for increased load we’re wasting money come the day there isn’t a bigger box to buy (something that is looking increasingly likely for many of us). All that money would have been better spent on buying racks of smaller boxes. It’s possible we can recover some of our losses by repurposing that big iron via virtualization rather than throwing it away (like all our previous big boxes) but of course, if that box dies it takes an awful lot of VM‘s with it.

Every time we assume we can keep all our data in a single memory or database (even if it’s a cluster) we’re embedding assumptions into our software that will be broken come the day we must partition across multiple memories or databases.

Each time we choose an algorithm that doesn’t easily partition or assumes a single memory/database we’re storing up trouble in our data and computational models.

In big monolithic systems it’s possible to create (by force) a never-fails environment which allows developers to ignore various edge cases. The move to a system built out of many separate parts makes failure almost impossible to avoid. This requires us to adjust our system design to take account of all those edge cases we previously ignored.

The time we spend gaining experience in building big monolithic systems has limited application when we switch to building distributed systems. We must learn new habits and adopt new modes of thought and that costs time.

In the worst cases, an organization’s processes, tools and departmental structure become heavily optimized for managing these big monolithic software and hardware systems such that it needs serious revision to cope with the move to horizontal, many box scaling. Typical problem areas include:

1. Monitoring – suddenly there’s a much greater number of machines to gather stats from. Existing gui representations mightn’t cope with such a large number.
2. Diagnosis – no longer does a single timestamp imply an order on events making analysis of logging information and root cause identification harder.
3. Deployment – previous methods simply break as the level of automation provided is inadequate for the number of machines and software components involved.
4. Testing – existing testing practices where everything can live on the developer’s desktop or in a single VM are no longer viable. There are too many moving parts and the convenience of isolation provided by testing at the desktop or in a single VM is lost.

I doubt threads will ever go away but learning to build and manage systems constructed in any of the following ways might be worthwhile:

1. Multiple communicating reliable processes on a reliable bus
2. Multiple communicating unreliable processes on a reliable bus
3. Multiple communicating unreliable processes on an unreliable bus

[ Where bus is typically a backplane or a network ]

Technorati Tags: architecture, distributed systems, process, scalability

One got addicted and the other ran away
Some settle down a familiar place
One lets go the wheel while the other one steers

One got the money that the other put away
Some held the world and the others couldn’t stay
A few just follow their dreams while the others stood clear
After all these years
After all these years

All These Years – Adema – Kill the Headlights

Technorati Tags: music, personal

Bobby Woolf penned a great article:

Clients often want to build only an ESB because that involves a technology challenge without the need for messy business requirements. Building just an ESB becomes an IT field of dreams, where IT builds an ESB and then hopes some SOA will come along and use it. Such an ESB-oriented architecture loses the benefits of SOA. It does not create business value. In fact, it incurs cost without reaping immediate benefit.

There in black and white is what happens when the focus becomes purely technological – no (business) value. The answer is an ESB, now what’s the question? A few years back we were saying:

The answer is WS-Deathstar, now what’s the question?

Before that it was:

The answer is an application server, now what’s the question?

Still further back we had:

The answer is a database, now what’s the question?

Silver bullet after silver bullet, when will we learn? Not for as long as we pursue technology for technology’s sake (death by technology). No doubt it’s rewarding for many a geek, fun for sure but it’s appropriate and innovative application of technology to a problem that matters (vision followed by implementation reality). Deploying a piece of technology on the basis that one might use or benefit from it’s presence in the future is simply foolish. It goes against the YAGNI principle for starters.

INATT is heard in the SOA realm partly because some parties are attempting to move beyond the endless hype curve/silver bullet delusion. Andrew McAfee appears to have been fortunate having clients that don’t suffer from death by technology. However judging by the experiences of Mr Woolf and myself not everyone is immune. So let’s talk tech but please only in the context of achieving a greater goal (unproven cost reduction promises don’t count). By way of example, consider this quote from Vogels:

Growth is core to Amazon.com’s business strategy, and that has had a significant impact on the way we use technology: growth through more categories, a larger selection, more services, more buying customers, more sellers, more merchants, more developers, increasing the different access methods, and expanding delivery mechanisms. The impact has been on many areas: larger data sets, faster update rates, more requests, more services, tighter SLAs (service-level agreements), more failures, more latency challenges, more service interdependencies, more developers, more documentation, more programs, more servers, more networks, more data centers. A large part of Amazon.com’s technology evolution has been driven to enable this continuing growth, to be ultra-scalable while maintaining availability and performance.

Technorati Tags: business, enterprise, soa, technology