Pragmatic Dictator

Jim Waldo writes:

My own conclusion is that system design is really a matter of technique, a way of thinking rather than a subject that can be taught in a particular course. It might be possible to build a program that teaches system design by putting students through a series of courses that hone their system design skills as they move through the subject matter of the courses. Such a series of courses would, in effect, be a formalized version of the apprenticeship that is now the way people acquire their system design technique…..

…..Even worse than not being visible to the customer, work done on designing the system is not visible to the management of the company that is developing the system. Even though managers will pay lip service to the teaching of The Mythical Man Month, there is still the worry that engineers who aren’t producing code are not doing anything useful. While there are few companies that explicitly measure productivity in lines-of-code per week, there is still pressure to produce something that can be seen. The notion that design can take weeks or months and that during that time little or no code will be written is hard to sell to managers. Harder still is selling the notion that any code that does get written will be thrown away, which often appears to be regression rather than progress.

In such an environment lip service often extends to technical strategy as well.

Some of the more common software development mistakes I’ve seen…..

Ignoring the triangle - The triangle represents a trade-off between three core elements of software delivery - resource, product (features, non-functionals, quality) and schedule. One can only ever control two elements, the third being determined by the decisions regarding the other two. So if one wishes to dictate product and schedule, sufficient resource must be made available to complete the task in the allotted time. If one wishes to dictate product and resource, then the schedule cannot be limited. It is simply “as long as it takes”. And if one wishes to dictate resource and schedule, then product features, quality etc must be traded away to allow completion of development within the time allotted.

It’s amazing how often organisations attempt to dictate all three elements and are then surprised when a project gets messy. Of course, development processes have evolved in recognition of this trade-off - agile for example is great for prioritising, dropping features and getting something useful out the door in a resonable timeframe with limited resources.

Heroic efforts - these are a bad sign. A regular pattern of projects turning into mad hack-fests, saved by some apparently super-talented individual(s) is indicative of broken processes. One step in addressing this problem involves an honest surgery immediately after the project to determine root causes (e.g. inadequate risk management) of the meltdown and methods of prevention for future projects (e.g. regular risk review and identification of appropriate mitigations).

In the very worst cases, management actively encourages such heroism via recognition and reward. Worshipping this kind of carnage and supposed miracle recovery is tantamount to approving bad project management. Note that well-intentioned management can unknowingly drive this behaviour. From McConnell’s Rapid Development:

Some managers encourage heroic behaviour when they focus too strongly on can-do attitudes. By elevating can-do attitudes above accurate and sometimes gloomy status reporting, such project managers undercut their ability to take corrective action. They don’t even know they need to take corrective action until the damage is done. As Tom DeMarco says, can-do attitudes escalate minor setbacks into true disasters.

No Risk Management - One can never predict or spot all the risks but there are some obvious ones that get missed over and over. For example, we’re building a piece of software that relies on a component we’ve not used before. This is a big risk, one that can be mitigated by writing a test-harness or simulation of the way in which we plan to use the component.

The simulation should include realistic load, failure conditions, maintenance etc and should be as close to the beginning of the project as possible to surface any issues early (we cannot afford to wait until final QA or deployment testing). There can be no shirking here because should our chosen component fail, we will need these tests in place so we can validate potential replacements as quickly and easily as possible.

Waterfall Agile - We’re supposedly “doing” agile but one or more of the following are true:

1. There’s a fixed deadline, with fixed features and fixed resources.
2. All Negotiation/Trade-off is done prior to project commencement with no review between sprints.
3. All sprints have been planned out in advance right up to the release date with no spare time.
4. There are no risks to manage (because there aren’t any apparently).
5. No-one is entertaining the idea of unknowns.
6. When a sprint doesn’t deliver as anticipated, outstanding work is simply crammed into the remaining sprints.

From On Designing and Deploying Internet-Scale Services (James Hamilton - Windows Live Services Platform):

We have long believed that 80% of operations issues originate in design and development, so this section on overall service design is the largest and most important. When systems fail, there is a natural tendency to look first to operations since that is where the problem actually took place. Most operations issues, however, either have their genesis in design and development or are best solved there. Throughout the sections that follow, a consensus emerges that firm separation of development, test, and operations isn’t the most effective approach in the services world. The trend we’ve seen when looking across many services is that low-cost administration correlates highly with how closely the development, test, and operations teams work together.

Yep, I’m a firm believer too….

One more chance to feel alive,

One more chance to wind up dead,

One more chance to climb so high,

One more chance to bust my head,

One more chance to get on top,

One more chance to make a change,

One more chance for shit to give,

One more chance to even the blame…….

Saliva - One More Chance - Blood Stained Love Story

A couple of false economies software development indulges in:

1. It’s quicker for me to write the code than explain the design to someone else.
2. Automated deployment will have to wait until we have more time.

Number one costs a software development team in a number of ways:

• The career development of other members of the team is slowed - if one never discusses design how does one expect to obtain good designers or architects?
• The team’s development capacity is reduced - essentially projects bottleneck around the uncommunicative heroic individual.
• The team’s effectiveness is reduced - project load cannot be divided efficiently because individuals have skills in narrow areas limiting the breadth of work they can perform.
• Team morale is damaged with other developers feeling left out, unfulfilled and unable to influence project decisions

Number two yields costs including:

• We save on some development time but the cost is re-surfacing in staff-hours required to perform the deployment.
• An increasing number of mistakes that extend deployment time or breaks releases.
• We save development time once and pay the price for that saved time with each and every deployment.
• The cost of each successive deployment increases because the system’s size is growing.
• As each deployment takes ever longer, the gap between releases is likely to increase.

No one would deliberately drive at night with their headlights off. It’s obvious why, we might make it to our destination but we’ll have run down a few pedestrians, bounced off some curbs, hit a lamp-post or two and slid into a few ditches. Were we to continue this way, our car would quickly turn into a useless wreck.

Driving with the headlights on allows us to see ahead, plan and anticipate a little, to think. In turn, our journey is more pleasant, the car lasts a lot longer and there’s much less risk of a fiery end.

Yet many companies drive with their headlights off when developing software. Silly deadlines with a non-negotiable set of features, fixed resource and no time to think. The result? A tangled mess of systems with zero-architecture, huge legacy, horrible brittleness and poor availability. And that most desired property of quick delivery is lost as it takes longer and longer to do even the most simple things.

There’s no substitution for prior thought and realistic planning. Yet so many eschew it whilst complaining about the results.

Technorati Tags: development, engineering, software

I mentioned a while back that one could exploit DNS to ease some of the common static configuration issues around hostnames, ports etc. What follows is a simple outline solution, we’ve moved a long way beyond this at Betfair but the details will have to remain secret for now (sorry).

Let’s assume that we have several different releases in testing at any one time such that we wish to segment our development/testing systems into separate enclaves (each handling a separate release) and may wish to add more enclaves over time. Assume also that production is an enclave in its own right.

Firstly we define a set of logical hostnames that refer to the significant components of our system such as databases, file servers etc. Other elements such as webservers are probably independent and not referenced from other parts of the system and thus do not need names. These logical hostnames are what feature in our configuration files and do not need to change from enclave to enclave because we are going to use DNS to map from these logical hostnames to real physical machines.

Thus we want is a separate namespace for hosts in each of these enclaves so as to prevent leakage. To that end we map each namespace onto a separate domain within our DNS setup.

[Note our DNS setup would typically consist of a set of servers that maintain records for our own internal domains and possibly forward other requests for say external web address to other servers.]

Each enclave therefore has:

1. A separate namespace represented as a unique domain
2. A set of services deployed onto physical machines
3. A mapping from logical machine names to physical machine names (or IP addresses)
4. A collection of configuration files all referencing logical machine names

Each domain (namespace) contains the logical to physical mapping of machines for its associated enclave. Each domain can be a separate zone and is thus kept in a separate file read by our DNS master. This allows us to maintain a template file which can be quickly edited to create a new domain (namespace). Thus whenever we wish to create a new enclave we setup a new zone, containing the definition of a new domain which is the namespace for that enclave.

To actually resolve a logical hostname we must ensure that it is concatenated with the domain appropriate to the enclave’s namespace. Before discussing options, note that each machine will be allocated to an enclave and must be configured accordingly which we can exploit to our advantage:

1. Simple configuration - ensure that the application has access to the domain to concatenate. This could be done via command-line argument but better is to source it from a well-known file on the machine which could be setup as part of allocating it to an enclave.
2. Default search domain - any name not fully qualified has the default search domain appended to it. This default is typically part of the resolver configuration of the operating system and again can be setup as part of allocating a machine to an enclave.

Missing from the above is the handling of ports which might change from one enclave to the next. This can be tackled with a similar logical/physical mapping approach but must be based on the use of DNS SRV records rather than simple hostname mappings. The JDK provides little help out of the box for querying these records so something like dnsjava will be required.

Technorati Tags: distributed systems, dns, testing

I commented in my previous post on the fact that working with message brokers can lead to tensions where we’re forced into letting the broker do more than we’d like.

I suspect that the bigger the list of features provided in a single broker the greater the probability we’ll be forced to comply with (and attempt to work around) undesirable models and behaviours that aren’t core to our requirements. Thus under certain circumstances a specific solution that addresses one core problem whilst asserting a minimum of additional constraints can be more attractive than a jack-of-all-trades solution.

It also occurred to me that brokers might not be the only piece of software to exhibit this characteristic. RPC systems tend to suffer similar problems having a tendency to tightly bind endpoint addressing, transport and argument marshalling. I also wonder if these issues contribute to some of the backlash against the behemoth that is an RDBMS implementation. For example whilst we might like our RDBMS to look after our data, we might not wish to put up with the querying, concurrency, deployment and management models it also asserts even with a myriad of configuration options.

Technorati Tags: architecture, design, technology

Say the word messaging to a subset of developers and for some reason the immediate knee-jerk is to assume that means using some kind of message broker (Tibco, ActiveMQ or whatever). Utter the term “asynchronous communication” and that is typically equated to messaging and thus also implies use of a message broker.

I find this strange because messaging is possible via a myriad of methods including carrier pigeon, pony express, tcp, multicast, http and many other transports. As for supporting asynchronous operations well that’s governed by the API(s) provided by the transport. In fact this is only partially true because a lower-level synchronous API can be wrapped in an additional layer to produce an asynchronous API. Transports and layering are often seen together, for example it allows us to introduce support for reliable delivery if the underlying transport is not robust (often it isn’t).

There’s no denying that all (or most) of these features (asynchronous APIs, messaging, reliable delivery) are provided in various of the message broker implementations but rarely in the form of a set of composable layers. The preferred approach is usually a myriad of configuration options leaving us at the mercy of the vendor to provide and support just the right combination of configuration possibilities to match our design challenge.

The indivisible nature of these brokers hampers us in other ways too. It can be difficult to use the broker purely for messaging without being forced to work with say its models for routing control and security. I’m sure this is appealing to the vendors but it doesn’t seem like such a great deal to me.

Technorati Tags: architecture, design, messaging

It never ceases to amaze me how often supposedly professional software people (developers, designers and architects) choose to trust in blind luck. They become rabbits in the headlights of pressure be it deadlines, expectation or lack of insight resorting to chucking code out the door as quickly as possible hoping it’ll lead to something good.

If it does lead to something good, how would we recognise it? If it’s going badly, how do we recognise that and formulate a new approach? What is the cost of taking such a speculative action? What’s the best thing that might come out of the action?

Many of these people would claim to be brilliant problem solvers and yet they are lacking grasp of some of the fundamentals.

Technorati Tags: engineering, process, software

Why do people still use static addressing in configuration files? Fixed hostnames or worse IP addresses?

These things make one’s life a nightmare when moving from one environment to another e.g. desktop to QA, QA to staging or staging to production.

With each transition, one must wade through all the relevant configuration files, find all these addresses and edit them. This creates many an opportunity for error such as missing one configuration variable or mistyping an address. It’s also a nightmare to maintain accurate documentation for all these scattered settings.

And yet this is so unnecessary if one exploits the abilities of DNS (and maybe Bonjour) properly. Just look at some of the cool stuff one can do. Better still most (all?) of it is supported in BIND.

Technorati Tags: architecture, distributed systems, systems

Mapreduce has taken some criticism from Stonebraker and DeWitt. I found this particular quote interesting:

we are amazed at the hype that the MapReduce proponents have spread about how it represents a paradigm shift in the development of scalable, data-intensive applications

Personally I’ve not heard any such claim from the MapReduce community (and who is that anyway?).

I’ve always seen MapReduce as nothing more than a useful tool for processing massive amounts of file-based data in ad-hoc fashions. This ad-hoc requirement is significant to me because DBMS’en have a tendency toward organizing data into static structures:

The DBMS community learned the importance of schemas, whereby the fields and their data types are recorded in storage. More importantly, the run-time system of the DBMS can ensure that input records obey this schema. This is the best way to keep an application from adding “garbage” to a data set. MapReduce has no such functionality, and there are no controls to keep garbage out of its data sets. A corrupted MapReduce dataset can actually silently break all the MapReduce applications that use that dataset.

These static structures can make it difficult to change things in support of new modes of usage. For example, one doesn’t lightly change index terms within a DBMS especially for large amounts of data. Perhaps the breadth of MapReduce queries run at Google would make regular index changes essential hence they chose to avoid a DBMS approach.

I wonder just how much the authors assumed about the way things are done at Google and what kind of “queries” they run. Consider that GFS which apparently underpins MapReduce is focused on append-only, not the sort of thing one sees in the DBMS world which accounts for updates and inserts amongst other things.

Many eyes in the industry have indulged in the bad habit of seeing the DBMS as the data storage equivalent of a Swiss Army Knife. It is the universal hammer for all data storage and analysis nails regardless of the actual requirements. Could Stonebraker and DeWitt have gotten caught up in this “classic mistake”? Surely not given what they’ve said in the past?

Recently I’ve had reason to go back over some data-structure theory, average times for insertion, deletion and search etc. I was reminded of a period around 1999 when I was implementing a Hyperchromatic Tree, a subtype of the basic red-black tree.

Hyperchromatic trees are lazily balanced i.e. balanced in the background rather than as part of the operation that changes the structure of the tree and they have some challenging locking disciplines to implement as well. They are designed to perform better under concurrent access than their ancestors.

Looking at my notes I was running a single test performing concurrent operations continuously for periods of 5 days against an instance containing 1 million entries on JDK 1.4. I suspect this period of my developer life taught me more about deadlocks and race conditions than any other.

Technorati Tags: concurrency, development, software

This just shouldn’t be a surprise. Frankly it’s closing the door long after the horse has bolted - check out Martin Odersky’s comments in this article from way back when. Who wants to bet that in spite of all this we still get closures in Java?

We make this mistake all the time with languages, frameworks, applications and operating systems. When will we learn the lesson I wonder?

Technorati Tags: philosophy, development, software

Bill ponders:

“Sometimes I wonder how a deployed n-tiered scale up monolith can be gradually refactored to a scale out model or run on scale out infrastructure. It’s well documented that companies like Amazon and Ebay have done just that, only how they did it tends to get left out of the slide decks. I suspect it involves thinking quite differently about what ‘good’ application code is….”

I think one reason for the absence of this information in slide decks is that it’s a competitive advantage, a barrier to entry. Mastering scale-out allows one company to handle more load and more customers whilst offering more function than another. Why teach other companies how to compete? For me though the real issue is that the transition is a major effort that defies easy explanation via slideware.

Makeup of a Monolith

The rough form of a monolithic system is a logical (maybe physical) n-tier with simple scale out in the places where it’s easy e.g. a stateless web tier. The remainder is typically a single database containing all data which is fronted by some kind of cache (some systems will also talk to legacy systems via various mechanisms and some will have an extra database or two). Some may view their database replication strategy/cluster to be horizontal scaling but to be the real deal would require explicit partitioning or sharding. All code has access to all of the data in the database and it’s typically not well controlled which leads to what I term the “integration via database” anti-pattern.

In essence to move to a scale out architecture requires us to partition up all the state we have held in the database (admitting there’s incentive to break up the middle tier to avoid it getting overly large and unwieldy). Sounds easy but of course is anything but because there’s just so much beyond “thinking quite differently about what ‘good’ application code is” to learn and change. One might expect that we build things up in horizontal slices, create some new infrastructure, port to the new infrastructure, the usual “upgrade the middleware” type approach but this doesn’t work due to lack of hindsight and the sheer size of the challenge. The antidote to this is to take one little step at a time.

Basic Approach

Instead of the “big scary re-architecture” we take vertical slices through the monolith, separating them out so that we can evolve each with a limited amount to consider. We push these pieces all the way through into deployment so as to expose all our processes and tools to this new type of entity. This is not easy but:

1. At least it’s being done against a small vertical slice which makes it a more manageable problem.
2. The transition cannot be managed in one big step, it will be necessary to manage old and new alongside each other so we need to get used to that.

Another option for learning how to work outside of the monolith is to implement some new set of features in this new style and drive that through the delivery chain. Obviously whilst this develops knowledge it’s not breaking down the existing monolith, so let’s move on to consider the basic steps (order is not fixed) for extracting a vertical element:

1. Identify some reasonable sized function or chunk of data to partition.
2. Define an interface which will wrap this function and/or data.
3. Move code and data behind the interface.
4. Modify other code to access code and data via the interface rather than e.g. direct access to the database.

Step (1) often requires us to examine both data and function. Ideally the interface we introduce in step (2) would look like a remote service. It may for some period of time still be part of the monolith and therefore local. In some cases we can’t make the interface remote initially because the refactor would be too complex so we must later come back and recast the interface as a remote service. What form does this new interface take? Ignoring religious issues it could be WS-*, REST, some other form of remote invocation (sometimes custom but not fine-grained method calls) or messaging.

The arrival of this new interface allows us to introduce behaviours such as asynchronous operation and eventual consistency into a controlled area of our codebase. It’s possible that the data we’ve wrapped behind the interface is still being drawn from the original database so we can also consider moving this data to it’s own separate storage mechanism (which may or may not be another database) and introduce sharding and partitioning.

Challenges

Some developers will fall back on “remoteness dogma” stating that remoteness is a performance inhibitor. Indeed they are right but scale is at least (if not more) important and becomes a key focus when we can’t buy a bigger box to make the monolith perform. For me the real issue at hand is the corruption of the mind that comes from thinking transactionally. In this world we focus on minimising the amount of time a transaction takes for fear of lock contention and blocking threads too long. We become obsessed with consistency, feeling compelled to do all work ahead of the actual transaction. This thinking naturally leads to optimising the execution paths heavily and eschewing remoteness. Key to tackling these issues is socialising use of asynchronous techniques and the fact that consistency is in the eye of the beholder (and determined by how often they can observe the relevant data).

During the recasting of a vertical slice into a standalone element it’s usually necessary to ensure it can cope with existing load (and a bit more). Establishing what the current load looks like can be challenging as monitoring and statistics are often centred around information available from the database, application servers, load-balancers and operating systems i.e. the infrastructure. This data is certainly useful but says little about what’s actually happening in the application code. A good way to address this problem and improve the lot of the operational teams is to introduce a programme of application instrumentation that will deliver appropriate statistics and high-level diagnostics.

Inevitably we will be adopting and/or building some new infrastructure but we must be careful how much we try to acquire and/or custom build in advance of real-world experience (see the hindsight link above). Fortunately there’s little that’s actually required up front other than some mechanisms for service location and statistics gathering so the impact of mistakes is usefully limited. Follow-on candidates might include messaging, deployment and security. Remember also that many vendors are still producing infrastructure suited to big-iron monolithic development and single data-centre environments (which can make resilience in the face of certain kinds of failure difficult).

Static configuration (per-machine or in tools) can be extremely troublesome containing a lot of URL’s, server addresses, machine names and database references. All of these items need changing at each stage from development, through testing, QA, staging and production. The move toward a more distributed approach will only make this worse as it creates a need to copy and tweak more configuration on more machines. It’s important from the early stages to focus on eliminating as much of this as possible. In an ideal world a machine would be configured with at most its designated duty and environment (testing, development, production) obtaining everything else it needs from services in the environment.

Wrap Up

To go horizontal is extremely challenging and cannot be addressed by the typical re-architecting initiatives many companies indulge in. There’s too much to learn and change such that the only option is a slow, step by step, learn as you go transition that gradually chips away at the monolith. For Amazon it seems this transition has taken at least five years and over at eBay it looks like they started the transition sometime around 1999/2000.

Technorati Tags: architecture, distributed systems