The machine consists of parts, discrete elements that perform particular roles required to support the functional, operational and performance characteristics.

A design should be pure. It may or may not be easily understood depending upon the body of knowledge used in its construction. For example, a design intended to be distributed will contain many subtleties and patterns that most are not familiar with.

The challenge, of course, is to make this abstract machine a reality, create a physical realisation of our abstract processing machine.

During this process of creation it is important we continue to honour the structures in the design. For, if we do not, we create a gap between design and physical reality such that it can be difficult to relate one to the other. Over time these gaps will multiply and widen, such that the physical realisation of the abstract machine no longer looks anything like the original design.

When gaps develop, there will be problems of traceability as one can no longer relate what is being done in the physical realisation to the design. Knowledge retention becomes problematic as the design can no longer be annotated and maintained for reference. The code and build system are now the only truth, containing structures that have been eroded and become grey, ill-defined. A “big ball of mud” develops that no one adequately understands. Knowledge is held in the heads of various people and scattered in code comments and notes on bits of paper.

It becomes harder and harder to reason about the behaviour of the physical realisation impeding speed of evolution. This manifests as reduced development speed or slow resolution of operational issues in production amongst other things.

There are a collection of things we can do to inhibit the onset of this death spiral including:

• Develop and retain a naming convention that fits with the nouns and verbs used in the design.
• Ensure that deployment onto physical machinery is an intuitive mapping that fits the inter-relatedness of components in the design. This compels the design to express constraints in respect of locality and methods of communication. It must also define failure modes and recovery protocols. This may require creation of further layers of abstraction atop physical machinery e.g. lookup mechanisms and remote access constructs.
• Considered selection of supporting infrastructure that works with the design rather than forcing compromises such as false separations of data or code, fractures in naming conventions, tight coupling where the design requires the opposite or introduction of behaviours that interfere with desired performance characteristics.
• Look for situations where the physical realisation is complicated by the design. When this happens, some re-design is natural but care must be taken to ensure that this is not occurring because of inappropriate technology choice.

Of course, nothing is perfect and some compromises must be made to bring these abstract machines into reality. Nevertheless too many compromises bring on the death-spiral so it is important to carefully consider those made and limit the number that exist at any moment in the lifetime of the physical realisation.

A quick google, shows that he’s not that public of a personality, seemingly preferring to let his products do the talking, works for me! Nevertheless there are some interviews out there, including this one which features a quote that I feel fits well (and is far more succinct) with my previous post:

Q. What are the advantages of designing for one company? And the disadvantages? What are the particular characteristics of the set-up at Apple that has made the experience of working there rewarding for you?

A. It is pretty humbling when so much of your effectiveness is defined by context. Not only is it critical that the leadership of a company clearly understands its products and the role of design, but that the development, marketing and sales teams are also equally committed to the same goals. More than ever I am aware that what we have achieved with design is massively reliant on the commitment of lots of different teams to solve the same problems and on their sharing the same goals. I like being part of something that is bigger than design. There is a loyalty that I have for Apple and a belief that this company has an impact beyond design which feels important. I also have a sense of being accountable as we really live, sometimes pretty painfully with the consequences of what we do.

1. Many geeks are obsessed with using all the latest technologies and building ultra-clever software that isn’t practical to run assuming it ever gets delivered.
2. Many managers fail to prioritise. An organisation can only ever be focused on one thing effectively at any moment. That’s because there are always bottlenecks, cash, mindset, staff, office space, the list goes on. Best identify the single most important thing you can do and go for it with all you have. Back a single horse, not the entire field.
3. Many geeks are all about the process. They’re agile, they’re kanban, they’re lean, they’re waterfall, they’re hypothesis testing. All of them miss the fundamental, which is not the process but the mindset that underpins it. Following the process blindly makes one a valueless robot, a cog in the machine, rather than an active participant shaping destiny.
4. Many managers fail to understand that dates are aspirational. One must actively work towards a date by managing scope, risk and resource, even then, there are no guarantees. What’s the value of a date anyway? Nothing. What matters is what you have to offer and whether it’s enough.
5. Many geeks fail to understand that design is a creative, subjective discipline. Design patterns aren’t the answer nor is the do it all framework. It’s knowing what to apply and when which comes from broad experience (which means you need to know more than one language, platform and operating system and will need to have worked in a number of industry verticals).
6. Many managers are obsessed with cost reduction. Fact is that to do something to a particular standard in some reasonable amount of time is going to cost a certain minimum. That minimum is largely unknowable in advance so better to decide how much you are prepared to pay and work to that number. As soon as an overshoot looks probable it’s time to step back and re-think (maybe even can) the plan.
7. Many geeks are performance junkies, optimising every part of their system based on a set of metrics that have nothing to do with the real-world. In the worst cases, there are no metrics, no one has done any measurement and the statement that something is “too slow” is pure speculation.
8. Many managers treat those that work for them as interchangeable components with no feelings, no individual strengths or weaknesses, no motivations or concerns. They then wonder why staff retention is appalling, performance is horrid and they can’t ever seem to recruit the right people.

What’s missing is a collective focus on producing something worth a damn. Producing such a thing requires caring and provides challenge, profit, happiness, satisfaction and a whole lot more. So the first question to ask whether you’re a geek or a manager is, what are we trying to do? Natural follow-ups would be why and do we care?

Whilst you’re pondering that, hopefully you’ll realise that there shouldn’t be a fence or sides…

Given this state of affairs, we need to develop means for handling the fact that staff move on. Typically that focuses everyone on recruitment of suitable replacements that:

• Already possess the appropriate technical skills
• Already have the relevant experience
• Already have the relevant mindset

The key word there is already. It presents us with a number of problems:

1. An equivalently skilled individual will almost certainly move on themselves, because they’ll soon figure out there’s no more opportunity.
2. There are only so many suitably skilled people available.
3. Unless we have a bottomless pit of money, we can only access some fraction of the overall pool of available people.
4. Someone who has the mindset may not yet have developed the skills.

All these factors limit our ability to find the perfect replacement. It follows that we should widen the net a little, look for those that could be what we need with a little training. Training is likely to be a mixture of courses and on the job, with the latter dominating as what is learnt in the classroom must be put into real practice to bring progress and value.

Adopting such an approach brings many benefits including a maintained quality of work and a level of loyalty which translates into good staff retention which in turn reduces the amount of recruitment we must do. Cool, huh?

One last fact: Too many organisations make no serious effort at training, expecting the right people to just walk through their doors. They want someone else to take the burden e.g. universities and give nothing back in return. Such selfishness ultimately leaves them short of the skills they desire incurring costs.

Employment is give and take, give some training and development and you’ll take the rewards.

To call it a trading system was, at least architecturally, a gross lie, as in fact it was an everything system: payments, accounts, customer profile, reporting (yes, OLAP and OLTP on the same database, madness) and the bet engine.

So the programme broke down into two parts:

• Clean up and separate out the components
• Replace certain components with new implementations

The programme of work started about 18 months ago and if we deliver the entire roadmap there’s another 18 months to go.

Thus far we’ve separated the B2B elements out (yes, they were hanging off the side of the “trading system” as well) and put a new data delivery infrastructure in place with considerably reduced latency and increased reliability. We’ve also just about completed the moving of all reporting into our OLAP systems with real-time updates from the OLTP elements (we used to do reporting refresh every 24 hours with all the painful load spike issues that go with that). The other essential element has been to eliminate the intimate relationship between website and trading system (most website content should not live in the betting engine).

The next major step we’re focused on is the splitting out of customer and account handling. Once this is done we’ll be in the happy situation where we can introduce our new bet engine and run in parallel with the old one so a customer placing bets on markets in either engine continues to get a complete and accurate view of their position (as do our traders).

Our other major area of focus is the development of a new betting engine and a key innovation there will be that we don’t use RDBMS’en for storing that information. We maintain auditing trails and DR abilities but with a faster, far lighter weight solution that will cost much less than what we’re running now.

Some technical details:

• We’ve opted for a service-based implementation, mostly with RESTful interfaces and always with smart stubs. Fact is we have to do a distributed solution to support our regulatory requirements effectively and efficiently (PCI, FSA and various gambling authority needs).
• We’ve implemented a service lookup mechanism from scratch based on gossip algorithms. This allows us sophisticated load and failure management strategies tuned on a service by service basis. It also gives us scope for admission control.
• We’re building up a new multicast infrastructure to deliver updates from the bet engine to desktops, other systems etc in real-time.
• Our bet engine is partitioned such that we can up or down scale on demand via virtualisation (no we can’t use most forms of cloud infrastructure as that breaches a number of regulations).
• We’ve got some nice automated recovery protocols that make recovery from hardware or component failures straightforwards for operational staff. In essence, they replace the broken element and it automatically knits itself back into the system and supports an SLA for recovery. For example, we can say that a cache will contain all relevant data within 5 minutes assuming a certain set of constraints are met (failure recovery times are difficult to guarantee 100%).
• Everything is monitored including stubs, services and infrastructure. Our operational teams get to routinely use what’s being developed and be involved in the specification of the data generated and the writing of the manuals. We’ve standardised the protocols/methods of exposure for both the monitoring data and logging output.

You’ll notice I haven’t talked about languages used and such. That’s because it doesn’t matter as with our service-based approach we can use whatever suits us best on a per service basis. That’s a key part of our general engineering philosophy, “right tool for the right job”, we don’t do fashion, buzz or hype influenced work, the pragmatic, practical, effective and efficient space is where we’re focused.

• We’re adopting lean (or agile)
• We’re using Scala
• We’re doing large-scale

Dig a little deeper though and you’ll quickly find:

• The agile or lean they’ve adopted and trumpet is merely a fixed set of ceremonies such as time-boxed cycles, limiting in-flight work, pair programming and automated testing.
• The Scala code they’re writing is largely imperative with no use of any of the obvious functional aspects.
• The supposed large-scale is in fact a couple of thousand customers running on a single database with an operation count per minute of no more than 60.

These individuals and companies are paying lip-service, they haven’t learnt anything other than what the latest buzz is and they’re operating some parody of the real thing.

The fundamental problem is, they don’t do their research. They don’t actually examine what has been done previously and understand its fundamentals, apply it and sharpen it. This is the skill that I gained at university, not some set of buzz technologies or processes. This is the fundamental skill that allows us to progress, to learn, adapt and act.

[ There's a supreme irony in an individual or company claiming to be doing agile or lean and not being able to research, learn and adapt because they follow a fixed set of ceremonies. ]

Unfortunately this skill isn’t generally recognised as valuable in the IT industry, it’s rarely taught and not seen in requests for curriculum changes from business to universities. This sort of feedback is more typical:

…startups need graduates who can hit the ground running, who are proficient in PHP, Python and Ruby (among other modern programming languages), and who, ultimately, understand the practical side of software engineering as opposed to just the theoretical side which they learn at university.

Being proficient in a language does not make you good, it just means you can crank (poor) code fast. Further, understanding the practical side of software engineering means learning from what you do, analysing and adjusting. If we’re all so good at that, why do we repeatedly get burnt by the same classic mistakes?

Far too many within IT act on hype or pay lip-service, they don’t do research, they don’t adopt the basic disciplines of learning. Whilst that continues, progress is nothing but a pretence.

I’ve always assumed this was well understood yet many organisations adopt processes, approaches and structures that guarantee certain kinds of defects will be undiscovered for substantial periods of time. One of the more common faults is the separation of Development and Operations.

Each side has its own view of what’s important and what they’re responsible for:

• Operations more often than not seeks to own non-functional aspects (performance, stability, scalability etc).
• Development more often than not seeks to own the functional aspects (features).

Such a mindset often leads to a classic process mistake, issues with the functional aspects get dealt with early and all those linked to the non-functional and operational are left unsurfaced until last moment grand testing regimes (P&C, User Acceptance Testing) dig them out or worse, are discovered at the point of release into production.

The warning signs are usually there if only we paid attention to them:

1. Developers work in isolation building, deploying and configuring the components they develop in ways that suit them. It follows that deployment and configuration are not optimised for production and do not account for any hard won operational experience.
2. Operations staff demand huge handover documents be written by developers and passed over with the product. Inevitably the documentation fails to account for operational concerns (what would a developer know about operations?)
3. There are separate environments for the purposes of validating correctness and accuracy of handover documents. After all developers can’t be trusted to get the documentation right so it must be checked.
4. The development environments are ad hoc with no resemblance to production (certainly they aren’t a scale unit of production). Leading to large numbers of problems at release time: files can’t be found, configurations are broken and various versioning issues present themselves.

The antidote is relatively straightforward, all development activity should be performed in a production like situation. For example:

1. Deployment and configuration of software components under development should be routinely performed by operational staff. The result is early knowledge transfer and the documentation can now be written by those best able to produce it (operations staff, not developers).
2. Development environments should contain appropriate network topology. Often production setups contain segregated networks for security or availability reasons. Ensuring developers are exposed early to these issues means software is more likely to account for these demands.
3. Monitoring and logging infrastructure should be as per production and used routinely for debugging and capture of data relevant to testing (performance, failure etc)
4. Development environments should be scale units of production. This permits early production-like performance testing. This should be backed up with routine robustness testing e.g. to identify memory leaks early.

A typical reaction is for development and operations staff to say this cannot possibly work and will slow development to a crawl. They aren’t actually wrong but they’re missing a key insight:

If development has slowed to a crawl it’s an early warning of future production troubles.

For example, if deployment is taking too much effort and time, something needs tweaking, simplifying or automating. What we’ve done is best summarised by a proverb from Toyota (via Eric Ries):

“Stop production so that production never has to stop”.

We’ve created a feedback loop that highlights defects spanning all concerns (functional, non-functional and operational) early which keeps costs down.

Clearly, delivering a given feature will take a little longer as we must account for all aspects from functional through non-functional and operational. That’s acceptable because if we don’t cover all these aspects we’re asking for trouble in many forms including:

• If we cannot adequately monitor the performance of a newly delivered feature there’s a direct impact on customer experience. They will know before we do that something is broken which leads to irate phone calls, lost revenue etc.
• If we cannot adequately track the effect of a new feature on customer behaviours, we cannot evolve it appropriately.

Needless to say developing features in this fashion fits well with lean and agile approaches.

So the antidote is relatively straightforward and there are development approaches that fit well with what needs to be done. The toughest challenge remains though, effecting the necessary mindset shift to get it done. It ought to be a little easier with the rise of DevOps but notably there are early signs of trouble as has been seen with lean and agile adoption.

There are many who claim to know and practice each of these disciplines but most are paying only lip service, picking out the bits of process, mindset or tooling that suit them and ignoring the rest.

Sporting Index is right in the middle of making this tricky jump from Dev and Ops to DevOps, I’ll let you know how we get on.

• It takes an increasing number of staff just to deploy and run the system.
• Customers face an increasingly bad experience in terms of interaction, performance and stability.
• One spends more time refactoring than developing new features – although in many cases developers will simply not bother with this effort which accelerates the drop in quality for customers.
• The level of coupling increases impacting the integrity of the design and making future change more difficult.

I call this the “design-by-product anti-pattern”. There are a couple of things that cause it to manifest:

1. Absence of a real design prior to product/framework/library selection – most of those given the remit for design cannot construct proper abstractions that are adequately divorced from implementation. That is they do not understand the core entities and operations that exist within the domain they are building a system for. Thus when products/libraries/frameworks are selected there is limited structure to assist in evaluating their appropriateness.
2. These products are used because they are on the list of “company approved technologies”. The justification for the existence of such a list is that it “reduces cost” which it might well do if all one accounts for is licenses and product support. Unfortunately, the cost equation is not nearly so simple (see above re: costs).
3. A related problem to “company approved technologies” is hot or favourite technologies preferred by the development team regardless of their appropriateness for use in any particular design situation.

Any product/library/framework is created by an individual who has their own view of how their customers design their systems and builds APIs accordingly. In the worst cases these individuals design APIs in total isolation, focused on making them theoretically perfect (for some definition of perfect). If we as customers create designs that do not align well with the views of these individuals, the result will be costly as we force the two designs together. The cost is magnified for each additional conflicting product/library/framework design.

Loose coupling as the result of proper definition of roles and responsibilities is the only tool we have to allow for future design evolution. Poor selection of products/libraries/frameworks erodes this property and should be avoided otherwise death-march awaits.

1. Roles
2. Responsibilities
3. Coupling

Role

The basic justification for the existence of some api, interface or class. A summary of what it’s for. Just as importantly, the role defines what a particular entity is not for.

Responsibility

The things that some entity can do/knows in support of a role.

Coupling

An expression of the dependencies between roles. This property tells us a lot about the state of our design.

Two things that are heavily dependent upon each other might well be serving individual parts of a single role and thus should be consolidated. If everything ends up in a single role, it can suggest that the current approach to classifying behaviours is missing some factors.

Coupling can be temporal such that, for example, one entity cannot dispatch its responsibilities without the presence of another at the same time. This might indicate the need for some work on handling availability issues in a distributed system.

Limited coupling is a sign of cohesion, clarity in roles and responsibilities which can be indicative of a clean, maintainable design.

Platform Neutral

These basics apply regardless of the platform one chooses to develop upon. Roles, responsibilities and coupling apply just as well to service architectures, databases (tables and associated triggers and packages) and applications in Java, Scala, Clojure, C# or any other programming environment.

Warning Signs

It is very common for individual developers or development teams to allocate additional functions to existing elements of a design unthinkingly, thus eroding its quality. This manifests in many ways including:

1. Some element of the system becomes the source of all information in respect of e.g. configuration or the entirety of customer data.
2. A single cache contains all data regardless of its nature (e.g. customer, account details, market price).
3. Some element of the system must always be running otherwise nothing else works.
4. Some element of the system has functions that span many different bits of data (e.g. customer, account, market price).

Rule of Thumb

Any entity within a system should do only one thing and it should do it well (often credited as Unix Philosophy). This applies to everything from applications and products to services and individual classes.

• Functions to be performed
• Data to be discovered, consumed and produced
• Non-functionals (e.g. SLAs)

The abstractions are then rendered into the real-world using appropriate hardware, technologies, patterns and languages. A good design:

• Exhibits few exception cases
• Has logic and/or data located neatly and predictably
• Applies a small set of core constructs repeatedly
• Addresses operational needs
• Considers cost versus value delivered
• Is as simple as possible
• Has the minimum of implementation assumption

There are several key failing points in the design process:

• No adjustment in the face of implementation feedback – No design is complete or perfect. There will always be missed details leading to brittle code, complex corner cases or convoluted solutions. It is critical that we monitor our progress and adapt the design accordingly.
• No up front design – Design is the skeleton upon which we hang technology choices and code structure. In it’s absence we rapidly descend into a world of difficult to navigate code and costly constraints set by uninformed product choices.
• No care in following the design – A key element of design is to place the right things in the right places. Failing to do this at code time increases coupling, makes maintenance difficult and can impact both performance and scalability. Similar effects occur as the result of poor technology selection.

Design and implementation go hand in hand yet many of us lack awareness of where the boundary between these two elements lies. We don’t understand how these elements interact with each other or appreciate the impact of decisions we make in respect of one element on the other.