Pragmatic Dictator

Jonathan Ive (Apple’s Design High Priest) has been knighted. Those that succeed in influencing technology as much as Mr Ive are well worth paying attention to.

A quick google, shows that he’s not that public of a personality, seemingly preferring to let his products do the talking, works for me! Nevertheless there are some interviews out there, including this one which features a quote that I feel fits well (and is far more succinct) with my previous post:

Q. What are the advantages of designing for one company? And the disadvantages? What are the particular characteristics of the set-up at Apple that has made the experience of working there rewarding for you?

A. It is pretty humbling when so much of your effectiveness is defined by context. Not only is it critical that the leadership of a company clearly understands its products and the role of design, but that the development, marketing and sales teams are also equally committed to the same goals. More than ever I am aware that what we have achieved with design is massively reliant on the commitment of lots of different teams to solve the same problems and on their sharing the same goals. I like being part of something that is bigger than design. There is a loyalty that I have for Apple and a belief that this company has an impact beyond design which feels important. I also have a sense of being accountable as we really live, sometimes pretty painfully with the consequences of what we do.

I’m a CTO, the traitorous techie that turned to the dark side of management or the unrealistic techie that won’t face up to management reality, depending on your point of view. Who am I really? I’m the man with experience on both sides of the fence. I’m the one who sees that the grass is decidedly scorched regardless of which side of that fence you sit.

1. Many geeks are obsessed with using all the latest technologies and building ultra-clever software that isn’t practical to run assuming it ever gets delivered.
2. Many managers fail to prioritise. An organisation can only ever be focused on one thing effectively at any moment. That’s because there are always bottlenecks, cash, mindset, staff, office space, the list goes on. Best identify the single most important thing you can do and go for it with all you have. Back a single horse, not the entire field.
3. Many geeks are all about the process. They’re agile, they’re kanban, they’re lean, they’re waterfall, they’re hypothesis testing. All of them miss the fundamental, which is not the process but the mindset that underpins it. Following the process blindly makes one a valueless robot, a cog in the machine, rather than an active participant shaping destiny.
4. Many managers fail to understand that dates are aspirational. One must actively work towards a date by managing scope, risk and resource, even then, there are no guarantees. What’s the value of a date anyway? Nothing. What matters is what you have to offer and whether it’s enough.
5. Many geeks fail to understand that design is a creative, subjective discipline. Design patterns aren’t the answer nor is the do it all framework. It’s knowing what to apply and when which comes from broad experience (which means you need to know more than one language, platform and operating system and will need to have worked in a number of industry verticals).
6. Many managers are obsessed with cost reduction. Fact is that to do something to a particular standard in some reasonable amount of time is going to cost a certain minimum. That minimum is largely unknowable in advance so better to decide how much you are prepared to pay and work to that number. As soon as an overshoot looks probable it’s time to step back and re-think (maybe even can) the plan.
7. Many geeks are performance junkies, optimising every part of their system based on a set of metrics that have nothing to do with the real-world. In the worst cases, there are no metrics, no one has done any measurement and the statement that something is “too slow” is pure speculation.
8. Many managers treat those that work for them as interchangeable components with no feelings, no individual strengths or weaknesses, no motivations or concerns. They then wonder why staff retention is appalling, performance is horrid and they can’t ever seem to recruit the right people.

What’s missing is a collective focus on producing something worth a damn. Producing such a thing requires caring and provides challenge, profit, happiness, satisfaction and a whole lot more. So the first question to ask whether you’re a geek or a manager is, what are we trying to do? Natural follow-ups would be why and do we care?

Whilst you’re pondering that, hopefully you’ll realise that there shouldn’t be a fence or sides…

One simple management fact: Staff move on. There are many reasons this happens, not least of which is the lack of additional opportunity to progress and learn.

Given this state of affairs, we need to develop means for handling the fact that staff move on. Typically that focuses everyone on recruitment of suitable replacements that:

• Already possess the appropriate technical skills
• Already have the relevant experience
• Already have the relevant mindset

The key word there is already. It presents us with a number of problems:

1. An equivalently skilled individual will almost certainly move on themselves, because they’ll soon figure out there’s no more opportunity.
2. There are only so many suitably skilled people available.
3. Unless we have a bottomless pit of money, we can only access some fraction of the overall pool of available people.
4. Someone who has the mindset may not yet have developed the skills.

All these factors limit our ability to find the perfect replacement. It follows that we should widen the net a little, look for those that could be what we need with a little training. Training is likely to be a mixture of courses and on the job, with the latter dominating as what is learnt in the classroom must be put into real practice to bring progress and value.

Adopting such an approach brings many benefits including a maintained quality of work and a level of loyalty which translates into good staff retention which in turn reduces the amount of recruitment we must do. Cool, huh?

One last fact: Too many organisations make no serious effort at training, expecting the right people to just walk through their doors. They want someone else to take the burden e.g. universities and give nothing back in return. Such selfishness ultimately leaves them short of the skills they desire incurring costs.

Employment is give and take, give some training and development and you’ll take the rewards.

When I arrived at Sporting Index, three or so years ago, my early tasks included the planning of a programme of work to overhaul the existing trading system.

To call it a trading system was, at least architecturally, a gross lie, as in fact it was an everything system: payments, accounts, customer profile, reporting (yes, OLAP and OLTP on the same database, madness) and the bet engine.

So the programme broke down into two parts:

• Clean up and separate out the components
• Replace certain components with new implementations

The programme of work started about 18 months ago and if we deliver the entire roadmap there’s another 18 months to go.

Thus far we’ve separated the B2B elements out (yes, they were hanging off the side of the “trading system” as well) and put a new data delivery infrastructure in place with considerably reduced latency and increased reliability. We’ve also just about completed the moving of all reporting into our OLAP systems with real-time updates from the OLTP elements (we used to do reporting refresh every 24 hours with all the painful load spike issues that go with that). The other essential element has been to eliminate the intimate relationship between website and trading system (most website content should not live in the betting engine).

The next major step we’re focused on is the splitting out of customer and account handling. Once this is done we’ll be in the happy situation where we can introduce our new bet engine and run in parallel with the old one so a customer placing bets on markets in either engine continues to get a complete and accurate view of their position (as do our traders).

Our other major area of focus is the development of a new betting engine and a key innovation there will be that we don’t use RDBMS’en for storing that information. We maintain auditing trails and DR abilities but with a faster, far lighter weight solution that will cost much less than what we’re running now.

Some technical details:

• We’ve opted for a service-based implementation, mostly with RESTful interfaces and always with smart stubs. Fact is we have to do a distributed solution to support our regulatory requirements effectively and efficiently (PCI, FSA and various gambling authority needs).
• We’ve implemented a service lookup mechanism from scratch based on gossip algorithms. This allows us sophisticated load and failure management strategies tuned on a service by service basis. It also gives us scope for admission control.
• We’re building up a new multicast infrastructure to deliver updates from the bet engine to desktops, other systems etc in real-time.
• Our bet engine is partitioned such that we can up or down scale on demand via virtualisation (no we can’t use most forms of cloud infrastructure as that breaches a number of regulations).
• We’ve got some nice automated recovery protocols that make recovery from hardware or component failures straightforwards for operational staff. In essence, they replace the broken element and it automatically knits itself back into the system and supports an SLA for recovery. For example, we can say that a cache will contain all relevant data within 5 minutes assuming a certain set of constraints are met (failure recovery times are difficult to guarantee 100%).
• Everything is monitored including stubs, services and infrastructure. Our operational teams get to routinely use what’s being developed and be involved in the specification of the data generated and the writing of the manuals. We’ve standardised the protocols/methods of exposure for both the monitoring data and logging output.

You’ll notice I haven’t talked about languages used and such. That’s because it doesn’t matter as with our service-based approach we can use whatever suits us best on a per service basis. That’s a key part of our general engineering philosophy, “right tool for the right job”, we don’t do fashion, buzz or hype influenced work, the pragmatic, practical, effective and efficient space is where we’re focused.

Many in IT would have you believe that they’re progressive, constantly advancing and learning. They will provide examples such as:

• We’re adopting lean (or agile)
• We’re using Scala
• We’re doing large-scale

Dig a little deeper though and you’ll quickly find:

• The agile or lean they’ve adopted and trumpet is merely a fixed set of ceremonies such as time-boxed cycles, limiting in-flight work, pair programming and automated testing.
• The Scala code they’re writing is largely imperative with no use of any of the obvious functional aspects.
• The supposed large-scale is in fact a couple of thousand customers running on a single database with an operation count per minute of no more than 60.

These individuals and companies are paying lip-service, they haven’t learnt anything other than what the latest buzz is and they’re operating some parody of the real thing.

The fundamental problem is, they don’t do their research. They don’t actually examine what has been done previously and understand its fundamentals, apply it and sharpen it. This is the skill that I gained at university, not some set of buzz technologies or processes. This is the fundamental skill that allows us to progress, to learn, adapt and act.

[ There's a supreme irony in an individual or company claiming to be doing agile or lean and not being able to research, learn and adapt because they follow a fixed set of ceremonies. ]

Unfortunately this skill isn’t generally recognised as valuable in the IT industry, it’s rarely taught and not seen in requests for curriculum changes from business to universities. This sort of feedback is more typical:

…startups need graduates who can hit the ground running, who are proficient in PHP, Python and Ruby (among other modern programming languages), and who, ultimately, understand the practical side of software engineering as opposed to just the theoretical side which they learn at university.

Being proficient in a language does not make you good, it just means you can crank (poor) code fast. Further, understanding the practical side of software engineering means learning from what you do, analysing and adjusting. If we’re all so good at that, why do we repeatedly get burnt by the same classic mistakes?

Far too many within IT act on hype or pay lip-service, they don’t do research, they don’t adopt the basic disciplines of learning. Whilst that continues, progress is nothing but a pretence.