1. Roles
2. Responsibilities
3. Coupling

Role

The basic justification for the existence of some api, interface or class. A summary of what it’s for. Just as importantly, the role defines what a particular entity is not for.

Responsibility

The things that some entity can do/knows in support of a role.

Coupling

An expression of the dependencies between roles. This property tells us a lot about the state of our design.

Two things that are heavily dependent upon each other might well be serving individual parts of a single role and thus should be consolidated. If everything ends up in a single role, it can suggest that the current approach to classifying behaviours is missing some factors.

Coupling can be temporal such that, for example, one entity cannot dispatch its responsibilities without the presence of another at the same time. This might indicate the need for some work on handling availability issues in a distributed system.

Limited coupling is a sign of cohesion, clarity in roles and responsibilities which can be indicative of a clean, maintainable design.

Platform Neutral

These basics apply regardless of the platform one chooses to develop upon. Roles, responsibilities and coupling apply just as well to service architectures, databases (tables and associated triggers and packages) and applications in Java, Scala, Clojure, C# or any other programming environment.

Warning Signs

It is very common for individual developers or development teams to allocate additional functions to existing elements of a design unthinkingly, thus eroding its quality. This manifests in many ways including:

1. Some element of the system becomes the source of all information in respect of e.g. configuration or the entirety of customer data.
2. A single cache contains all data regardless of its nature (e.g. customer, account details, market price).
3. Some element of the system must always be running otherwise nothing else works.
4. Some element of the system has functions that span many different bits of data (e.g. customer, account, market price).

Rule of Thumb

Any entity within a system should do only one thing and it should do it well (often credited as Unix Philosophy). This applies to everything from applications and products to services and individual classes.

• Functions to be performed
• Data to be discovered, consumed and produced
• Non-functionals (e.g. SLAs)

The abstractions are then rendered into the real-world using appropriate hardware, technologies, patterns and languages. A good design:

• Exhibits few exception cases
• Has logic and/or data located neatly and predictably
• Applies a small set of core constructs repeatedly
• Addresses operational needs
• Considers cost versus value delivered
• Is as simple as possible
• Has the minimum of implementation assumption

There are several key failing points in the design process:

• No adjustment in the face of implementation feedback – No design is complete or perfect. There will always be missed details leading to brittle code, complex corner cases or convoluted solutions. It is critical that we monitor our progress and adapt the design accordingly.
• No up front design – Design is the skeleton upon which we hang technology choices and code structure. In it’s absence we rapidly descend into a world of difficult to navigate code and costly constraints set by uninformed product choices.
• No care in following the design – A key element of design is to place the right things in the right places. Failing to do this at code time increases coupling, makes maintenance difficult and can impact both performance and scalability. Similar effects occur as the result of poor technology selection.

Design and implementation go hand in hand yet many of us lack awareness of where the boundary between these two elements lies. We don’t understand how these elements interact with each other or appreciate the impact of decisions we make in respect of one element on the other.

P&C is one of those 80/20 tradeoffs. There’s always more one can do or measure or test, equally getting the basics in place will deliver substantial benefit. I’d go further and argue that without a solid grasp of the basics, one cannot easily determine what else beyond that might be required. Here then are the basics that I’ve found myself repeating over and over:

• Have an enquiring mind – anomalies are not to be ignored or dismissed on the basis of pure speculation. Determining root cause is essential to prevent surprises in production. Some recent examples:
  1. In one test we noticed that every so often we’d get a substantial blip in disk I/O on servers that should be processing entirely out of memory. Along with that blip there’d be a corresponding reduction in throughput, we could have ignored it, after all things sorted themselves out relatively quickly but we chose to investigate. All these servers were periodically running a cleanup job the developers were unaware of and had not factored into their capacity calculations. The implications for production would have been a regularly overloaded, badly performing website. We’ve since tuned the jobs, adjusted their schedules and increased our capacity to ensure we can always spread the load around enough to accommodate them.
  2. An examination of the distribution of load on the boxes behind our load-balancers revealed a higher than expected amount of variance in CPU and connections. A review of the application revealed that any particular user’s traffic is sticky to one box, unfortunate as it’s stateless, time for a code change. We also spent time looking at the monitoring infrastructure and discovered that in certain cases we’d get false reports of 100% CPU utilisation, that one will be fixed with an OS patch.
• Gather the right data – there’s no value in allowing oneself to be limited by what is easily available via some set of tools people are comfortable with. One tool we were using had an unreasonably low ceiling on the number and rate of samples it could handle such that any graphs it produced showed hardly anything of the true profile of e.g. CPU utilisation, memory consumption or I/O. Forming any opinion about system behaviour in respect of load was going to be an exercise in speculation. We junked the tool and are looking for a replacement, in the meantime we’ve fallen back to making use of low level performance counters which we can sample local to the machine and whack onto disk for later analysis via scripts, opensource tools etc.
• Design tests that support reasoning – One should indeed try and replicate production load behaviours to judge overall system behaviour. The challenge of such testing is that it can be difficult to relate performance data back to exactly what was going on during some period of a test and make a diagnosis or be confident of an improvement. There are a number of things we can do to improve the situation:
  1. Ensure tests are deterministic such that any given run can be compared against other runs. This isn’t as simple as it looks when e.g. you wish to gradually increase load at a fixed rate that is being produced by more than one box.
  2. Have tests produce sufficient logging that one can easily identify what was going on at particular points in the sampled data. Logging of course can actually affect test behaviour and that isn’t always desirable.
  3. Build additional tests that target particular user journey’s through the system. Doing this for all possible journey’s can be costly so it makes sense to focus on testing those which are most popular with users. These kinds of tests restrict the reasoning tree making analysis, diagnosis and solution identification much easier.
• Measure what customers care about – they don’t care about CPUs, I/O or memory, they worry about things like response times. It is important to focus on maintaining a quality user experience not endlessly improving system efficiency. Considering user factors such as response times stops us expending huge effort on CPU utilisation when we should be focusing on say, network I/O, browser performance or reducing the amount of data we push to the browser before a page can render.
• Beware of averages – it is very tempting to combine datasets via the use of averaging unfortunately such a practice can easily hide spikes that might be indicative of a problem. On more than one occasion an engineer has presented a graph that tracks the average CPU and a table that summarises min, avg and max. After which they’ve pronounced load testing was a success and yet they have no explanation for why the average is never more than 50% but the max is 100% and whether or not this is good or bad.




• More than load – excessive focus on measuring the effect of a particular load can make us blind to another important metric, resource cost per unit of work – these are the collection of tests and analysis that help us understand what to tune and how much to keep our appetite for boxes and bandwidth reasonable. One simple thing teams can do per sprint (assuming you’re agile, why wouldn’t you be?) is point a profiler at each component and look for the low hanging fruit that is poor algorithm selection or inefficient code (e.g. repeated scanning of lists where a hashmap would be better or repeatedly computing something that could be cached).

1. Making changes becomes increasingly expensive – make one small change and it spiders into changes across many other areas and gets into corners one least expects.
2. Replacing components of the system because for example they’re no longer supported, don’t perform adequately or can’t scale requires significant reverse engineering to understand dependencies etc.
3. It only takes one piece of the system failing to bring everything to its knees.
4. Isolating the root cause of a bug takes significant amounts of effort because it’s difficult to quickly eliminate large chunks of the system.

More often than not it’s believed (I’m guilty) these systems come into being through incompetence or indiscipline on behalf of the developers involved but I think there’s maybe another contributory factor: Much of the advice on design and architecture is couched in terms of design from scratch, there’s less guidance in regard to working with an existing architecture.

The result is that when developers start out building a system they have a lot of advice they can apply but as it grows, it becomes more difficult to apply the advice and discern what changes are appropriate, so the architecture unravels. Is there a way to avoid this unravelling? I believe there is and it’s derived from the process for fixing up an errant architecture.

These architectures have smells equivalent to the code-level examples Fowler discusses in his book on refactoring such as:

1. Some area of the system is too tightly coupled, making changes harder.
2. Some part of the system contains an assumption that there is only one resource of some type (e.g. a database) limiting scaling.
3. Many components of the system are reliant upon one key component being constantly available such that if it fails, nothing works.

Having identified these smells we need to perform appropriate cleanup which, for the list of examples above might include:

1. Placing additional APIs (interfaces) within the tightly coupled area of the system to reduce shared implementation knowledge and create well-bounded islands of data.
2. Introducing a resource discovery pattern to abstract away the assumption of a single resource at a single address.
3. Introducing concepts like acceptable staleness of data which allows caching for a period of time, eventual consistency which supports making updates and resolving the outcome at a later date or asynchronous operations.

It’s important to realise that in any substantial system we will be unable to eradicate a smell completely in a single update because it’s too risky. There will be many places in the code we might forget to patch up, a high likelihood we’ll miss something in testing, low probability we’ll get API designs exactly right etc. We must gradually introduce modifications over a period of time (months or even years) rather than perform significant rewrites. This isn’t as bad as it seems because no architecture is perfect for very long once it’s exposed to users. It also suggests that perhaps we need to focus on documenting techniques for gradual evolution of an architecture.

If we were to get better at spotting these architectural smells early (slight odour as opposed to horrific stench) and working to address them sooner than later it might be possible to avoid having a system’s architecture unravel, leading to something more sustainable.

Updated: to include additional commentary on APIs and perfection.

1. Cheaper operational costs.
2. Dynamic scaling in response to load spikes.
3. Roll-on, roll-off deployments for e.g. newspaper archive processing.

These platforms exist as the result of the investment of companies such as Amazon, Google and Microsoft in developing cost-effective infrastructure with system to administrator ratios of 2500:1 (whilst the average enterprise manages around 150:1 and inefficient properties manage maybe 10:1).

Key to allowing these infrastructures to be efficient and in turn deliver the benefits above is having applications architected such that:

1. They don’t require masses of administrator intervention when they go wrong.
2. They can be installed with minimal administrator effort because there’s no need to worry about tweaking URLs, IP addresses, database connections etc.
3. They readily support horizontal scaling e.g. because they contain an abstraction that can support sharding of data-storage.

In essence an application must be designed for zero administrator intervention and fully automated deployment. It should also have a variable workload component that magnifies the savings of the architectural properties above.

Strange then that many a developer expects to move their existing application, full of enterprise DNA (static configuration, vertical clusters, no horizontal scaling, high administration costs) to such an offering with minimal change. They even complain when it proves difficult because all those “enterprise features” aren’t present. Why does this happen?

I believe it’s because these developers have fundamentally misunderstood how cloud computing delivers its benefits. They see the cheap prices but don’t stop to consider where the cost saving comes from. Some of it is achieved by cloud platform vendors getting large discounts on huge hardware orders but a significant proportion comes from the fact that they don’t need to provide (via human resources or APIs) the sysadmin functions required for conventional hosting solutions.

Quite simply typical applications, their architectures and associated administration practices are not setup for cloud platforms. Some of them may be able to run on these platforms with sufficient hackery, brute force and associated cost. However if the motivation for a move to the cloud is merely to reduce kit costs one might well be better off looking for a cheaper conventional hosting solution.

In summary, making the best of the cloud requires that we take an architectural view, something that we’ve proven remarkably bad at over and over. Simply deploying an application unchanged to the cloud is unlikely to deliver much benefit.

• We’ll eliminate static wiring using Spring
• We’ll model everything as a service
• We’ll adopt test-driven development and make use of jMock
• We’ll build everything using a RESTful approach
• We’ll avoid using RPC in favour of messaging
• …….

Things will be so much better in this brave new world but……they won’t. The reason the codebase has got into a mess is because we failed to execute on important principles such as:

1. Take account of coupling and cohesion.
2. Be clear about people’s roles and responsibilities to avoid unqualified or inappropriate decision making.
3. Clarity and simplicity of roles and responsibilities in design elements.
4. Maintain modular, well-isolated code and conceptual integrity.
5. Avoid shared data-schemas or integration via the database.
6. Make the software testable and maintain the tests.
7. Select technology based on appropriate design work.
8. No broken windows.
9. Track and maintain appropriate metrics.
10. Review projects to identify and disseminate useful lessons to developers, architects and customers.
11. Account for the operational aspects of our software in requirements and design.
12. Review to ensure code aligns with appropriate design principles.
13. Surface, balance and mitigate risks.

It’s these principles and others that enable superior engineering which in turn delivers a good-quality, maintainable codebase. Any rewrite will end up a ball of mud just like it’s predecessor unless the style of engineering is adapted to incorporate principles such as these.

Some propose that frameworks can prevent mistakes, ensure a quality design and deliver testable code. I think experience suggests otherwise as we routinely (by accident or design) bend frameworks to fit some problem they weren’t really designed for leading to ugly, broken, poorly designed, brittle code. What would stop us doing it with new frameworks delivered as part of a grand re-write?

Should we successfully revise our engineering practices would we then have sufficient leverage to restructure our ball of mud into something nicer to work with? Maybe, maybe not but we might be better equipped to answer the question: re-write or re-factor?

• Deployment
• Packaging
• Configuration
• Monitoring
• Logging

Failing to address these elements is detrimental to core aspects of what we need to do from day one:

• Get changes out – ship a new feature, deploy an urgent bug-fix or make a tweak to handle a load-spike.
• Determine if things have started up and configured properly.
• Be sure things are still running right.
• Identify and react to problems quickly.
• Obtain data important to future architectural decisions.

Even in light of the above many of us are still tempted into leaving this until later by which time:

1. Our software will have grown substantially making it difficult and expensive to adapt when we do decide to address the operational issues.
2. We’ll be losing inordinate amounts of time on manual trouble-shooting and dealing with the consequences of human error (a key contributor to downtime and other problems).
3. Operations will likely have become tightly bound to whatever our software currently looks like such that when we start addressing the issues, we’ll break all their assumptions (and the tooling they built around them).

Some Specifics

Having configuration buried inside your binaries where it cannot be easily managed is an inconvenience. We don’t really want to have to do a whole new build just to change configuration settings (though one might want to do a re-deploy of the whole lot together to allow for audit-trails and have half a chance of having all boxes configured similarly at the same time).

When it comes to deployment and packaging it pays to adopt something akin to the xcopy install approach. Everything required is contained inside of the distribution with minimal external dependencies (necessary external dependencies should ideally be satisfied dynamically at runtime rather than with static configuration). Such an approach for desktop software would be unattractive but with servers and an imperative to automate installation it’s very attractive.

What about all those existing packaging systems such as rpm? Many of these mechanisms have a design assumption around a single version of something on a machine. This can inhibit fast rollback because rather than stopping one process and starting another one has to (in simple terms):

1. Stop a process.
2. Uninstall it’s binaries and dependencies.
3. Install the binaries for the old process and dependencies.
4. Start the other process up.

In some cases it will also be necessary to perform further configuration (did we back it up?), suddenly it’s looking like a lot of work to buy ourselves appropriate risk-mitigation for broken upgrades.

Monitoring often requires an amount of configuration which can make for a bootstrap problem where one needs monitoring to detect a configuration issue but the monitoring isn’t configured yet. Thus it can be useful to have some very simple monitoring based on a primitive that can run without explicit configuration such as multicast.

Important Step

These key operational elements should be accounted for early on in the design of system and grown alongside other functional aspects.^* There’s plenty of information on this topic publicly available including:

• Randy Shoup – eBay Marketplace Architecture.
• Dan Pritchett – Architecture Quality: Operational Manageability.
• Wayne Fenton – Operational Stability in The Next Generation Web World (a variation of the talk above).
• Michael Isard – Autopilot: Automatic Data Centre Management.
• James Hamilton – Designing and Deploying Internet Scale Services.

* Initially implementation can be simple scripts but at some point it becomes necessary to take a more serious approach in respect of tools and infrastructure development. This means investing in properly skilled architects and engineers, performing appropriate testing etc.

• Brittleness in the face of failure.
• High cost solutions.
• Poor scaling properties.
• Disappointment as the expectations of the requirements source aren’t met.

Consider a system where we have two (network) hops to an observer and one hop to the initiator of an action (assuming uniform network latency for each hop). Potentially for every two actions there will be a single observation. Thus each observation of the system is out of date by the time it reaches the observer.

Administrative actions can suffer similar problems, in that it could take several hops for the request to arrive at the system. A user may be only one hop away and could be performing many operations in the time it takes for one of our actions to reach the system. For example if we wish to block a user, whilst our request is in transit they might perform several operations.

Things are made worse by network failures which can further delay or prevent execution of an action and slow down the rate of updates to an observer.

How then do we account for these troubles when specifying requirements? By qualifying them with appropriate SLA’s. In the example above, appropriate SLA’s might include:

• Time for propagation of an administrative action.
• Maximum acceptable time after the action is triggered for a user to be blocked.

SLA’s such as the above:

1. Help us to identify appropriate solutions (e.g. do we need to pay for multiple independent routes between data-centres).
2. Allow us to make appropriate use of asynchronous operations and eventual consistency.

Since SLA’s have significant impact on the way in which a requirement will be implemented it is essential to perform appropriate expectation management, discussing and communicating the implications with the requirements source, they cannot be solely the domain of techies. Remember also that in many situations customers prefer availability over consistency.

• As a codebase grows it must be split up along functional boundaries, and spread across multiple processes. More code equals more processes and more machines to run them on.
• More customers, means more load and requires more machines to handle it.
• More data means more storage and more processors to chew through it.

Now let’s see how many machines some of the big players are running and what infrastructure they’re talking about:

TicketMaster have at least 3000 machines and have built Spine to help them manage configuration of their infrastructure.

eBay have built a custom deployment tool (Roller), logging infrastructure, configuration management for their software services, messaging software and more. They’re running around 15000 machines across four geographical locations.

Microsoft have built a custom deployment, configuration and monitoring infrastructure called Autopilot focused on many thousands of machines. In fact we’re talking hundreds of thousands.

Google are dealing in a million or more machines and expending effort on software to handle staged, automatic upgrades. Of course they’ve already built GFS, Chubby etc.

Twitter have moved beyond the half-dozen or so machines they used to have to “a lot of servers” (hundreds?) and are seemingly still hiring operations staff but have built a custom queue server.

Facebook have at least 10000 webservers, 800 MemcacheD instances and 1800 MySQL instances. They’ve built a custom configuration-serving infrastructure, management and monitoring tools. They also contribute to MemcacheD and have built Cassandra and Thrift. They also appear to be busy building their own optimized webservers and a replacement for squid.

Amazon have tens of thousands of servers (surely more?) and have constructed Dynamo, S3, EC2, SQS etc.

A few tentative conclusions:

1. It would seem that by the time a website has moved into the thousands of boxes it will have had to address configuration and monitoring. Which suggests development efforts started before this threshold (perhaps at a couple of hundred boxes?)
2. As the machine count moves towards the tens of thousands, automated deployment becomes essential and there’s a need to develop more service-specific infrastructure.

So if the fixed methods of SOA are thrown out and technology is not the solution, how do we build a system? Let’s first consider some of the things we’d like from our architecture:

1. Avoid integration via the database – otherwise data coupling will cripple us
2. Support for granular updates – taking down the whole system is not desirable
3. Fast rollback of changes – in case an update breaks
4. In-production testing – there’s no substitute for real traffic in tests
5. Minimal shared resources such as storage – so should there be an outage, impact is minimised
6. Horizontal scaling – more boxes equals more power
7. Support for scalable development – dev teams should be able to act in isolation most of the time
8. Support for appropriate CAP tradeoffs – making everything consistent can be bad for availability

Although we wish to avoid coupling via the database, the reality is that our code still requires access to the data in some form or another. The best we can do under this circumstance is to limit the amount of code that directly accesses the data. We achieve this by vertically slicing (as opposed to horizontal sharding) our data and consolidating the code that is most closely related to it (e.g. performs updates) into a single encapsulated unit. All other access to the data must go via the code element of its associated unit (note that one needn’t always go to a unit for the data, it’s perfectly acceptable to cache).

In this way we limit the impact of data-schema changes to it’s associated unit, other parts of the system need not be concerned but there’s still some work to do. If the code within a unit were to be co-located within all processes containing code that wishes to make use of it, we’d need to restart all those processes when we wish to deploy a new version of that code (for whatever reason). Such a deployment model also encourages several bad habits:

1. Ignoring the remoteness of the data – it’s hidden behind some form of interface and it’s tempting to attempt to hide failure behind that interface
2. Focus on synchronous method calls – it’s natural for a developer to write synchronous method calls when the code being called looks local (note that method calls can support asynchronous behaviours)

To avoid these issues, we deploy each unit in it’s own process accessed via some network endpoint that dependants use to interact with it thus:

• Each unit can now easily be allocated it’s own independent storage, apply it’s own sharding policy etc.
• The network endpoint can support multiple protocol versions or we can opt to terminate multiple network endpoints onto a unit, a powerful primitive for supporting several versions of a remote interface simultaneously.
• The network endpoint can be terminated onto some form of load balancer or custom routing implementation (which might be part of the code within the unit itself perhaps because it’s P2P based) facilitating horizontal scaling, hot upgrades, A/B testing, in-production tests etc.
• Each unit can be assigned to a development team and much work can be done independently of development efforts elsewhere, making for less contention in development.
• Each unit can implement whatever CAP tradeoff makes sense.

If we arrange for the network endpoint of each unit to be discovered dynamically at runtime we gain the ability to move our units around (e.g. for DR reasons) and have means for our system to dynamically knit itself together reducing configuration issues. Such an arrangement can also make it easier to deal with ordered startup issues (where some set of things must be available before others).

Of course it’s not all good news, we will have to manage our desire for ACID guarantees because many of the mechanisms (such as two-phase commit) for achieving this in a distributed system are fraught with problems. Fortunately, people have been thinking about this for a while. We’ll also have to take care of the fallacies but even this has some positive aspects as failure and upgrade in some cases can be considered the same (noting that abstractions for message passing, failure detectors and the like can be implemented in many languages, not just Erlang).

So what remoting approaches might we use? REST/http, WS-*, RMI, CORBA, messages, custom protocol – whatever is suitable for our situation (noting that some choices impact the means by which we can handle evolution of protocols etc). What guidelines might we follow in determining how to split our code and data? There are a number of different approaches including:

1. Considering similarities in consistency, availability and partitioning (CAP) requirements
2. Data access localities
3. Data relationships
4. Jurisdictional requirements
5. Roles and responsibilities (at coarser level than OO)
6. Features (e.g. recommendations)
7. Business processes
8. Constituent elements of an overall business process

Most systems likely require a combination of these rather than one fixed approach, taste and gut instinct count for a lot. And what might we call these units I speak of? I prefer to call them services as do a few other people but there’s no doubt that’ll be confusing, have to think of something else…….

[1] I know that Steve might well argue otherwise.