Advancing The Practice

Silver bullets have plagued software engineering for many years. Time after time, there’s some hot new thing that will solve all our problems. Then, a short while later, it’s pronounced a grand failure and never heard from again. More often that not, that pronouncement is the harbinger of the next hype cycle. Talk about wasteful!

The infection is deep, it’s thoroughly penetrated our social structures:

  • Many conferences are about all the cool things we might do with this or that. All the latest and greatest, the hot new things. Tutorials and workarounds for common problems.

  • There are many online tutorials, plenty of write-ups of this or that collection of patterns with new or old names, a pile of process, technology, tools, language and library documentation.

  • Case studies and whitepapers abound with positive stories of big money savings and so on.

  • There are advocates for everything. You gotta be agile or lean, you must use this or that DD, you have to be using this language etc.

The infection is further assisted by corporate behaviours such as the jealous guarding of knowledge via various intellectual property protection mechanisms. Somewhat ironic as without a level of public disclosure, it’s difficult to identify, attract and retain the kind of talent they require to sustain them1.

Just as importantly, the silver bullet is no respecter of history. Often the supposedly new thing is not, in any meaningful way, different from or better than what has come before.

Too rarely do we see real, unbiased, evidence-based discussion of suitability or comparison against other options including the historical. Something as substantial as an examination of the success and failure of a thing across time and a meaningful number of real-world applications2.

In absence of such material, it’s no surprise that the silver bullet virus thrives. How could one possibly build a meaningful body of knowledge that moves the discipline forward? How could any engineer make meaningful judgements of what might be appropriate or otherwise in any relevant arena (e.g. process, design, testing, debugging, operations) except from personal experience or fortune in finding the right mentors? How could silver bullets be seen for what they are?

Can we do better? I believe so.

Firstly we must get to grips with being honest about failure. Folks in the operational disciplines have a long tradition of publishing postmortem material (aka disaster porn). They’re also focused on improving methods for identifying and eliminating their shortcomings. We must adopt similar disciplines in software development covering architecture, process, development practices and more. However we must go further than even that because knowledge work is fundamentally about humans and the systems they’re embedded in. Failures in practice at this level must also be examined.

Second of all, I believe we must actually put the science back into computer science3. Quite simply, we’ve lost the ability to do our research properly, hypothesise adequately and analyse the results of our choices appropriately. Not all that we do in software development can be treated this way but we can get much better at evaluating our methods, tools, design decisions, patterns and such. We have many opportunities to run, document and publish the results of experiments4. Publishing is critical whether we succeed or fail (remember the above, we must be honest). Without it we cannot peer review or test for repeatability. Nor do we create a body of knowledge, a documented history.

If we can combine this honest, critically evaluated history with an appropriate discipline of research and experimentation I believe we can avoid wasteful duplication, enable learning and create genuine progress. Further the silver bullets formed from an ignorance of history and insubstantial claims can be seen for what they are and consigned to the trash can5.

Update History

22/3/14 Added a tweeted reference to Gilb on architecture that appeared at just the right moment.


  1. These company practices inhibit the building of portfolios with substance (architecture diagrams, key bits of code and such). In their absence, CV’s loaded with buzzwords and valueless references to the largely unseen are somewhat inevitable. From this intangible material, companies are supposed to identify talent. 
  2. There are of course exceptions such as the body of work Google publishes
  3. That means ensuring each undergrad understands the scientific process and it’s applications within their work. 
  4. For some ideas on how this might be done for architecture, see Gilb speaking or skim the slides
  5. No doubt, some will be quite happy to continue on as they are. At least though, these cargo cult engineers will be more easily discernible. Others can then make informed decisions in respect of hiring and such. 

On What Makes a Service

For me, the fundamental tradeoff is in transport costs, data encapsulation and computational cost. These factors interact with each other such that too much or too little of one can be indicative of a sub-optimal choice elsewhere.

Transport Cost

Building services increases the significance of network performance (we’re talking latency but bandwidth contributes). Network interactions are typically measured in milliseconds so we must ensure that each round-trip incurred is doing something worthwhile. A modern microprocessor will get through an awful lot of work in milliseconds so creating a service where the dominant cost is in transport is likely a mistake.

Too little means: Potentially we didn’t need the cost at all. Careful consideration of the value of the interaction should be undertaken. It may be that the model is acceptable because it facilitates other useful properties such as a single endpoint for wide dissemination of key information.

Too much means: Excessive wait time at the caller. Whilst we may benefit in respect of other factors, the overall experience will be lacking. It may have been quicker and cheaper to generate/source the data locally (and when we say locally we mean, on-box, a trip to a database cluster is incurring transport cost).

Interactions: Computational cost, when dominating the transport cost makes for a better tradeoff. Data encapsulation might result in bulky or chatty data transfers with negative impact on cost.

Data Encapsulation

The creation of a service allows us to break off a chunk of data and manage it alongside a related body of code. This is a particularly appealing characteristic when coping with a previously constructed monolith that has now got unwieldy in terms of schema and/or code complexity.

Too little means: Manageable structures (code and data) but no net reduction in complexity of reasoning across the codebase.

Too much means: A mini-monolith that is still difficult to maintain or enhance. At this end of the spectrum one likely still gains and perhaps might use such a service as a stepping stone to further decomposition.

Interactions: Transport cost will potentially vary alongside the amount of data present. Examples: the vast majority of the stored data is being passed over the wire with high latency or there’s a lot of data but the code digests it into something reasonable in respect of transport costs.

Related aspects: A lot of data can be valuable to a lot of third parties, balancing the right level of interest with correct data partitioning makes for manageable load. Computational cost in processing the data can also be a factor.

Computational Cost

Tuning a single codebase that contains many different workloads is a challenge. Separating out the major players makes options for tailoring architectural approach available. I define the computational cost of a service as all the work done (so that’s CPU and I/O), directly in response to a request and in background processing (e.g. Consuming events from elsewhere and digesting them or other actions to maintain the data) to produce useful data or action.

Too little means: Each request/action doesn’t produce sufficient value to justify a round-trip to get/deliver the results. In essence, the client/recipient would be better doing the computation itself saving on latency.

Too much means: Response time suffers although it would likely be the same (or worse) regardless of the location of the code. Additional architectural work is likely required such as selecting a new scale-out approach.

Interactions: Encapsulation somewhat dictates the amount of computation that must be performed locally due to coupling. In turn this influences value to other consumers of accessing the service which has implications for load and transport costs.

Related aspects: Code complexity contributes to computational cost such that when the cost is high, one might consider breaking something up into additional services. Before taking such a step though, an honest assessment of code cleanliness and choice of algorithm should be made.

Tweaking The Tradeoff

At this level, API design counts for a lot, it’s a key means of influence. Elements worthy of consideration include:

  1. Locality – Means for placing data and computation close to consumers influences transport costs. Services can be layered things, a core of data and code with edges that can be moved “closer” (network-wise) to consumers. This is often effected via caching and leases either explicitly in the API or under the covers. In some cases, one would build primitives such as state push into the API so consumers can build caches.

  2. Round-trip or hop-length reduction – By placing connectivity locally using early termination of SSL handshakes, avoiding chatty APIs or batching primitives.

  3. Asynchronous work – Trading a level of immediacy for better response time and possibly more scale/locality options. Application of the CAP Theorem via eventual consistency or the use of callbacks (such as webhooks) are typical.

  4. Storage design – There are interesting developments afoot in respect of storage hierarchies such as use of flash backed by spinning media. It’s been a while since the storage scene has been so volatile (no pun intended) but it can have a significant impact on computational cost for certain loads.

Edit History

15/3/14 – Published and minor addition to tweaks regarding hop-length.
16/3/14 – Updated to include some explicit treatment of I/O.

R on Mavericks

I’ve been wanting to play with R for some time. Installation can be tricky depending on your setup and I ended up going back to source code. Below, some notes on how to make it happen should you need to tread the same path.

  1. Install the latest Xcode and associated command-line binaries from Apple Developer.
  2. Download and compile from source GNU Readline. Configure it to Install in /usr/local. Before compiling make sure to apply this change.
  3. Download and compile from source libpng (use one of the green source code links). Configure it to Install in /usr/local.
  4. Install GFortran binaries from the GCC Wiki.
  5. Install MacTex to permit generation of R PDF manuals. Make sure to add /usr/texbin (the standard install location, documented here) to your shell’s path.
  6. Download and build R from source-code.
  7. Optionally install RStudio.

If you’ve chosen to have R installed in a private location (as opposed to system wide/for all users which is effected by invoking make install):

  1. R should be configured (via .Renviron) with a user-specific location in which to store packages.
  2. If you’ve installed RStudio, it will be necessary to tell it where R is located via a shell environment variable:

    export RSTUDIO_WHICH_R=/usr/local/bin/R

Need

It’s not about profit – making money does not help customers.

It’s not about cost-saving – saving money does not help customers.

It’s not about shareholders – pleasing shareholders does not help customers.

It’s not about features – delivering features does not help customers.

It’s not about testability – that something is tested does not help customers.

Satisfying a need does help customers. Amongst other things it might make their lives easier, make something possible, educate them or entertain them. It engages them, enthrals them, creates emotion within them. From all of this, many good things will come.

If you are developing a system in absence of a focus on satisfaction of needs, you’ve lost already. First question then:

Who are your customers?

And if you think the only customers are those paying for what you build, you’ve lost once again. In fact, you’ve signed your own happiness away.

On The Practice of Design

Technology is not architecture or indeed design, it is a means for implementing a design. Various technologies (e.g. languages or frameworks) will be more or less compatible with implementing a specific design.

Design is an abstract exercise. It becomes constrained by our own choices which can include using existing technology or creating anew. By default, it should not be constrained, this is closer to the ideal. The more constraint exerted by technology the less ideal things are likely to be. Less ideal can be acceptable, there are always cost limits and such but it should never exist without consideration of the consequences.

Some argue that design cannot be an abstract exercise at all because real-world considerations demand otherwise. Performance is often cited as being too significant to ignore. Are they right?

The nature of performance in a system can be generalised into a set of guiding principles. In the case of computing, there’s a well known hierarchy driven by locality (starting with the fastest component):

  1. Register-based CPU instruction
  2. On CPU cache access
  3. Off CPU cache access
  4. Main memory access
  5. I/O (network, conventional disk, SSD)

Jeff Dean and others have expressed this in a table of “Numbers every programmer should know”. The performance relationship amongst these components is sufficient guidance for design work. Clearly, incrementing a number across a network connection is something to be avoided (though there are ways to make this work). It would generate significant chatter as would naive distribution of an OO design.

So to answer the question: Performance is too important to ignore in a design but the amount of consideration required is no more than other aspects such as coupling and cohesiveness.

Apple decide what they want to build first then create and/or select the technologies they need leading to great products. They ask themselves how do we make this idea, this concept we have in mind, real? This is the point at which technology becomes relevant. NASA, when set the moon-shot challenge created the technologies they needed to deliver the end result over a period of years. They iterated on engine, flight control and many other aspects to get the ultimate embodiment. 37Signals ended up creating Rails, embodying a new way of building product to deliver their vision.

The best designs start out as concepts or ideas and are largely un-constrained by technology (there are limits of course, e.g. a phone must have certain components possessing certain properties). They retain their elegance, a sense of style and taste. Designs that are forced to fit with early, uninformed technological choices are likely to be brittle and die.

Developers have a bad habit of selecting tools and technology well ahead of consideration of a problem and potential design approaches. History is littered with examples of the consequences, balls of mud and expensive “surprise” project failures that should have been “easily dispatched” because of this or that silver-bullet technology.

There is nothing harmful in the general discussion of technology tradeoffs, it’s what leads to useful guidance such as that of Jeff Dean above. It also makes sense in one’s early career to work with a variety of technologies to help gain an understanding of tradeoffs and patterns that work or don’t. However, excessive technology fixation is destructive for quality design work.

One can certainly design from a technology driven perspective (choose your language, frameworks etc and constrain your design to fit them) but that won’t be good enough for a moonshot, a class-leading product or a high-quality solution.