A quick google, shows that he’s not that public of a personality, seemingly preferring to let his products do the talking, works for me! Nevertheless there are some interviews out there, including this one which features a quote that I feel fits well (and is far more succinct) with my previous post:

Q. What are the advantages of designing for one company? And the disadvantages? What are the particular characteristics of the set-up at Apple that has made the experience of working there rewarding for you?

A. It is pretty humbling when so much of your effectiveness is defined by context. Not only is it critical that the leadership of a company clearly understands its products and the role of design, but that the development, marketing and sales teams are also equally committed to the same goals. More than ever I am aware that what we have achieved with design is massively reliant on the commitment of lots of different teams to solve the same problems and on their sharing the same goals. I like being part of something that is bigger than design. There is a loyalty that I have for Apple and a belief that this company has an impact beyond design which feels important. I also have a sense of being accountable as we really live, sometimes pretty painfully with the consequences of what we do.

1. Many geeks are obsessed with using all the latest technologies and building ultra-clever software that isn’t practical to run assuming it ever gets delivered.
2. Many managers fail to prioritise. An organisation can only ever be focused on one thing effectively at any moment. That’s because there are always bottlenecks, cash, mindset, staff, office space, the list goes on. Best identify the single most important thing you can do and go for it with all you have. Back a single horse, not the entire field.
3. Many geeks are all about the process. They’re agile, they’re kanban, they’re lean, they’re waterfall, they’re hypothesis testing. All of them miss the fundamental, which is not the process but the mindset that underpins it. Following the process blindly makes one a valueless robot, a cog in the machine, rather than an active participant shaping destiny.
4. Many managers fail to understand that dates are aspirational. One must actively work towards a date by managing scope, risk and resource, even then, there are no guarantees. What’s the value of a date anyway? Nothing. What matters is what you have to offer and whether it’s enough.
5. Many geeks fail to understand that design is a creative, subjective discipline. Design patterns aren’t the answer nor is the do it all framework. It’s knowing what to apply and when which comes from broad experience (which means you need to know more than one language, platform and operating system and will need to have worked in a number of industry verticals).
6. Many managers are obsessed with cost reduction. Fact is that to do something to a particular standard in some reasonable amount of time is going to cost a certain minimum. That minimum is largely unknowable in advance so better to decide how much you are prepared to pay and work to that number. As soon as an overshoot looks probable it’s time to step back and re-think (maybe even can) the plan.
7. Many geeks are performance junkies, optimising every part of their system based on a set of metrics that have nothing to do with the real-world. In the worst cases, there are no metrics, no one has done any measurement and the statement that something is “too slow” is pure speculation.
8. Many managers treat those that work for them as interchangeable components with no feelings, no individual strengths or weaknesses, no motivations or concerns. They then wonder why staff retention is appalling, performance is horrid and they can’t ever seem to recruit the right people.

What’s missing is a collective focus on producing something worth a damn. Producing such a thing requires caring and provides challenge, profit, happiness, satisfaction and a whole lot more. So the first question to ask whether you’re a geek or a manager is, what are we trying to do? Natural follow-ups would be why and do we care?

Whilst you’re pondering that, hopefully you’ll realise that there shouldn’t be a fence or sides…

To call it a trading system was, at least architecturally, a gross lie, as in fact it was an everything system: payments, accounts, customer profile, reporting (yes, OLAP and OLTP on the same database, madness) and the bet engine.

So the programme broke down into two parts:

• Clean up and separate out the components
• Replace certain components with new implementations

The programme of work started about 18 months ago and if we deliver the entire roadmap there’s another 18 months to go.

Thus far we’ve separated the B2B elements out (yes, they were hanging off the side of the “trading system” as well) and put a new data delivery infrastructure in place with considerably reduced latency and increased reliability. We’ve also just about completed the moving of all reporting into our OLAP systems with real-time updates from the OLTP elements (we used to do reporting refresh every 24 hours with all the painful load spike issues that go with that). The other essential element has been to eliminate the intimate relationship between website and trading system (most website content should not live in the betting engine).

The next major step we’re focused on is the splitting out of customer and account handling. Once this is done we’ll be in the happy situation where we can introduce our new bet engine and run in parallel with the old one so a customer placing bets on markets in either engine continues to get a complete and accurate view of their position (as do our traders).

Our other major area of focus is the development of a new betting engine and a key innovation there will be that we don’t use RDBMS’en for storing that information. We maintain auditing trails and DR abilities but with a faster, far lighter weight solution that will cost much less than what we’re running now.

Some technical details:

• We’ve opted for a service-based implementation, mostly with RESTful interfaces and always with smart stubs. Fact is we have to do a distributed solution to support our regulatory requirements effectively and efficiently (PCI, FSA and various gambling authority needs).
• We’ve implemented a service lookup mechanism from scratch based on gossip algorithms. This allows us sophisticated load and failure management strategies tuned on a service by service basis. It also gives us scope for admission control.
• We’re building up a new multicast infrastructure to deliver updates from the bet engine to desktops, other systems etc in real-time.
• Our bet engine is partitioned such that we can up or down scale on demand via virtualisation (no we can’t use most forms of cloud infrastructure as that breaches a number of regulations).
• We’ve got some nice automated recovery protocols that make recovery from hardware or component failures straightforwards for operational staff. In essence, they replace the broken element and it automatically knits itself back into the system and supports an SLA for recovery. For example, we can say that a cache will contain all relevant data within 5 minutes assuming a certain set of constraints are met (failure recovery times are difficult to guarantee 100%).
• Everything is monitored including stubs, services and infrastructure. Our operational teams get to routinely use what’s being developed and be involved in the specification of the data generated and the writing of the manuals. We’ve standardised the protocols/methods of exposure for both the monitoring data and logging output.

You’ll notice I haven’t talked about languages used and such. That’s because it doesn’t matter as with our service-based approach we can use whatever suits us best on a per service basis. That’s a key part of our general engineering philosophy, “right tool for the right job”, we don’t do fashion, buzz or hype influenced work, the pragmatic, practical, effective and efficient space is where we’re focused.

• We’re adopting lean (or agile)
• We’re using Scala
• We’re doing large-scale

Dig a little deeper though and you’ll quickly find:

• The agile or lean they’ve adopted and trumpet is merely a fixed set of ceremonies such as time-boxed cycles, limiting in-flight work, pair programming and automated testing.
• The Scala code they’re writing is largely imperative with no use of any of the obvious functional aspects.
• The supposed large-scale is in fact a couple of thousand customers running on a single database with an operation count per minute of no more than 60.

These individuals and companies are paying lip-service, they haven’t learnt anything other than what the latest buzz is and they’re operating some parody of the real thing.

The fundamental problem is, they don’t do their research. They don’t actually examine what has been done previously and understand its fundamentals, apply it and sharpen it. This is the skill that I gained at university, not some set of buzz technologies or processes. This is the fundamental skill that allows us to progress, to learn, adapt and act.

[ There's a supreme irony in an individual or company claiming to be doing agile or lean and not being able to research, learn and adapt because they follow a fixed set of ceremonies. ]

Unfortunately this skill isn’t generally recognised as valuable in the IT industry, it’s rarely taught and not seen in requests for curriculum changes from business to universities. This sort of feedback is more typical:

…startups need graduates who can hit the ground running, who are proficient in PHP, Python and Ruby (among other modern programming languages), and who, ultimately, understand the practical side of software engineering as opposed to just the theoretical side which they learn at university.

Being proficient in a language does not make you good, it just means you can crank (poor) code fast. Further, understanding the practical side of software engineering means learning from what you do, analysing and adjusting. If we’re all so good at that, why do we repeatedly get burnt by the same classic mistakes?

Far too many within IT act on hype or pay lip-service, they don’t do research, they don’t adopt the basic disciplines of learning. Whilst that continues, progress is nothing but a pretence.

Vendors can claim they are doing it and support it, developers can claim they do it, use it or implement it. There are a bunch of ready examples: Agile, XP, SOA and REST. Naming something makes it easy to ignore or forget its underpinnings, the elements that deliver value.

As a martial artist, I’m familiar with this pattern of behaviour: various people claim to practice and teach authentic Silat, Karate, Kung Fu, Escrima and so on. Inevitably some of them are exposed as pretenders. One of the more notable martial artists, Bruce Lee was sufficiently concerned about this that he gave serious consideration to leaving his approach to martial art (Jeet Kune Do) unnamed^*.

Is it worth naming things? Might we be better served by making our knowledge, approaches and philosophies visible for others without naming them to adopt or not as they see fit? Would it reduce the number of valueless certifications, buzzword cv’s and endless wars over which way is the way and who’s doing it right?

* Jeet Kune Do (1997) ‘Actually, I never wanted to give a name to the kind of Chinese gung fu that I have invented, but for convenience sake, I still call it “Jeet Kune Do”. However, I want to emphasize that there is no distinction between jeet kune do and any other kind of gung fu, for I strongly object to formality, and to the idea of distinction of branches.’

we are amazed at the hype that the MapReduce proponents have spread about how it represents a paradigm shift in the development of scalable, data-intensive applications

Personally I’ve not heard any such claim from the MapReduce community (and who is that anyway?).

I’ve always seen MapReduce as nothing more than a useful tool for processing massive amounts of file-based data in ad-hoc fashions. This ad-hoc requirement is significant to me because DBMS’en have a tendency toward organizing data into static structures:

The DBMS community learned the importance of schemas, whereby the fields and their data types are recorded in storage. More importantly, the run-time system of the DBMS can ensure that input records obey this schema. This is the best way to keep an application from adding “garbage” to a data set. MapReduce has no such functionality, and there are no controls to keep garbage out of its data sets. A corrupted MapReduce dataset can actually silently break all the MapReduce applications that use that dataset.

These static structures can make it difficult to change things in support of new modes of usage. For example, one doesn’t lightly change index terms within a DBMS especially for large amounts of data. Perhaps the breadth of MapReduce queries run at Google would make regular index changes essential hence they chose to avoid a DBMS approach.

I wonder just how much the authors assumed about the way things are done at Google and what kind of “queries” they run. Consider that GFS which apparently underpins MapReduce is focused on append-only, not the sort of thing one sees in the DBMS world which accounts for updates and inserts amongst other things.

Many eyes in the industry have indulged in the bad habit of seeing the DBMS as the data storage equivalent of a Swiss Army Knife. It is the universal hammer for all data storage and analysis nails regardless of the actual requirements. Could Stonebraker and DeWitt have gotten caught up in this “classic mistake”? Surely not given what they’ve said in the past?

Should be good….

Technorati Tags: blog, technology

Clients often want to build only an ESB because that involves a technology challenge without the need for messy business requirements. Building just an ESB becomes an IT field of dreams, where IT builds an ESB and then hopes some SOA will come along and use it. Such an ESB-oriented architecture loses the benefits of SOA. It does not create business value. In fact, it incurs cost without reaping immediate benefit.

There in black and white is what happens when the focus becomes purely technological – no (business) value. The answer is an ESB, now what’s the question? A few years back we were saying:

The answer is WS-Deathstar, now what’s the question?

Before that it was:

The answer is an application server, now what’s the question?

Still further back we had:

The answer is a database, now what’s the question?

Silver bullet after silver bullet, when will we learn? Not for as long as we pursue technology for technology’s sake (death by technology). No doubt it’s rewarding for many a geek, fun for sure but it’s appropriate and innovative application of technology to a problem that matters (vision followed by implementation reality). Deploying a piece of technology on the basis that one might use or benefit from it’s presence in the future is simply foolish. It goes against the YAGNI principle for starters.

INATT is heard in the SOA realm partly because some parties are attempting to move beyond the endless hype curve/silver bullet delusion. Andrew McAfee appears to have been fortunate having clients that don’t suffer from death by technology. However judging by the experiences of Mr Woolf and myself not everyone is immune. So let’s talk tech but please only in the context of achieving a greater goal (unproven cost reduction promises don’t count). By way of example, consider this quote from Vogels:

Growth is core to Amazon.com’s business strategy, and that has had a significant impact on the way we use technology: growth through more categories, a larger selection, more services, more buying customers, more sellers, more merchants, more developers, increasing the different access methods, and expanding delivery mechanisms. The impact has been on many areas: larger data sets, faster update rates, more requests, more services, tighter SLAs (service-level agreements), more failures, more latency challenges, more service interdependencies, more developers, more documentation, more programs, more servers, more networks, more data centers. A large part of Amazon.com’s technology evolution has been driven to enable this continuing growth, to be ultra-scalable while maintaining availability and performance.

Technorati Tags: business, enterprise, soa, technology

There’s one statement that I might contend with just a little:

"The WS-* community really believes that developers are too stupid to be allowed to manage themselves. Developers have to be told what to do and kept from getting their grubby little hands all over the network protocols because they can’t be trusted to make the right choices."

The WS-* community may well see things this way but I think there’s at least one other possibility that would place the fault elsewhere. Given that an awful lot of developers are heard to utter sentences like…

"I don’t want to be bothered with the nitty gritty details of network protocols, threads, persistence etc I just want to write my business logic"

…perhaps it’s natural to expect that various entities will construct technologies like WS-*. Developers are seemingly pushing responsibility elsewhere, placing their fate in others hands and paying the price. Perhaps they should be more careful what they wish for?

Habitual behaviours are the worst because one can accept the need for change, even be making it but also be completely unaware of subconscious behaviours from the old world getting in the way or continuing to drive us.

Self awareness then and being aware of our behaviours (consciously knowing what we are doing) is very important if we are to make key cultural change. It requires mentoring, questioning and motivating. Alas some people simply won’t be able to make the change. Most important of all, these changes happen sloooooowly. Although the more people who “get it” the more amplified the change gets and hopefully the quicker the remainder learn new habits.

So I’m left to ponder, just how often have we really changed things? How often have we fooled ourselves into believing we’ve achieved a revolution when actually we’ve managed little more than a slight evolution. If everyone can understand something so quickly is that not because it fits with existing understanding and behaviours? If everyone gets to carry over their creature comforts (gets what they are used to) have we not re-asserted old world thinking?

Technorati Tags: business, philosophy, technology