Pragmatic Dictator

Mapreduce has taken some criticism from Stonebraker and DeWitt. I found this particular quote interesting:

we are amazed at the hype that the MapReduce proponents have spread about how it represents a paradigm shift in the development of scalable, data-intensive applications

Personally I’ve not heard any such claim from the MapReduce community (and who is that anyway?).

I’ve always seen MapReduce as nothing more than a useful tool for processing massive amounts of file-based data in ad-hoc fashions. This ad-hoc requirement is significant to me because DBMS’en have a tendency toward organizing data into static structures:

The DBMS community learned the importance of schemas, whereby the fields and their data types are recorded in storage. More importantly, the run-time system of the DBMS can ensure that input records obey this schema. This is the best way to keep an application from adding “garbage” to a data set. MapReduce has no such functionality, and there are no controls to keep garbage out of its data sets. A corrupted MapReduce dataset can actually silently break all the MapReduce applications that use that dataset.

These static structures can make it difficult to change things in support of new modes of usage. For example, one doesn’t lightly change index terms within a DBMS especially for large amounts of data. Perhaps the breadth of MapReduce queries run at Google would make regular index changes essential hence they chose to avoid a DBMS approach.

I wonder just how much the authors assumed about the way things are done at Google and what kind of “queries” they run. Consider that GFS which apparently underpins MapReduce is focused on append-only, not the sort of thing one sees in the DBMS world which accounts for updates and inserts amongst other things.

Many eyes in the industry have indulged in the bad habit of seeing the DBMS as the data storage equivalent of a Swiss Army Knife. It is the universal hammer for all data storage and analysis nails regardless of the actual requirements. Could Stonebraker and DeWitt have gotten caught up in this “classic mistake”? Surely not given what they’ve said in the past?

Met Kyle a few years back on a consulting gig down in Nashville. Been nagging him not quite ever since to blog and finally he’s seen sense.

Should be good….

Technorati Tags: blog, technology

Bobby Woolf penned a great article:

Clients often want to build only an ESB because that involves a technology challenge without the need for messy business requirements. Building just an ESB becomes an IT field of dreams, where IT builds an ESB and then hopes some SOA will come along and use it. Such an ESB-oriented architecture loses the benefits of SOA. It does not create business value. In fact, it incurs cost without reaping immediate benefit.

There in black and white is what happens when the focus becomes purely technological - no (business) value. The answer is an ESB, now what’s the question? A few years back we were saying:

The answer is WS-Deathstar, now what’s the question?

Before that it was:

The answer is an application server, now what’s the question?

Still further back we had:

The answer is a database, now what’s the question?

Silver bullet after silver bullet, when will we learn? Not for as long as we pursue technology for technology’s sake (death by technology). No doubt it’s rewarding for many a geek, fun for sure but it’s appropriate and innovative application of technology to a problem that matters (vision followed by implementation reality). Deploying a piece of technology on the basis that one might use or benefit from it’s presence in the future is simply foolish. It goes against the YAGNI principle for starters.

INATT is heard in the SOA realm partly because some parties are attempting to move beyond the endless hype curve/silver bullet delusion. Andrew McAfee appears to have been fortunate having clients that don’t suffer from death by technology. However judging by the experiences of Mr Woolf and myself not everyone is immune. So let’s talk tech but please only in the context of achieving a greater goal (unproven cost reduction promises don’t count). By way of example, consider this quote from Vogels:

Growth is core to Amazon.com’s business strategy, and that has had a significant impact on the way we use technology: growth through more categories, a larger selection, more services, more buying customers, more sellers, more merchants, more developers, increasing the different access methods, and expanding delivery mechanisms. The impact has been on many areas: larger data sets, faster update rates, more requests, more services, tighter SLAs (service-level agreements), more failures, more latency challenges, more service interdependencies, more developers, more documentation, more programs, more servers, more networks, more data centers. A large part of Amazon.com’s technology evolution has been driven to enable this continuing growth, to be ultra-scalable while maintaining availability and performance.

Technorati Tags: business, enterprise, soa, technology

Elliotte Rusty Harold with an interesting commentary on the REST vs WS-* war.

There’s one statement that I might contend with just a little:

"The WS-* community really believes that developers are too stupid to be allowed to manage themselves. Developers have to be told what to do and kept from getting their grubby little hands all over the network protocols because they can’t be trusted to make the right choices."

The WS-* community may well see things this way but I think there’s at least one other possibility that would place the fault elsewhere. Given that an awful lot of developers are heard to utter sentences like…

"I don’t want to be bothered with the nitty gritty details of network protocols, threads, persistence etc I just want to write my business logic"

…perhaps it’s natural to expect that various entities will construct technologies like WS-*. Developers are seemingly pushing responsibility elsewhere, placing their fate in others hands and paying the price. Perhaps they should be more careful what they wish for?

So many times the hardest part of moving things forward is effecting change in culture and habitual behaviours.

Habitual behaviours are the worst because one can accept the need for change, even be making it but also be completely unaware of subconscious behaviours from the old world getting in the way or continuing to drive us.

Self awareness then and being aware of our behaviours (consciously knowing what we are doing) is very important if we are to make key cultural change. It requires mentoring, questioning and motivating. Alas some people simply won’t be able to make the change. Most important of all, these changes happen sloooooowly. Although the more people who “get it” the more amplified the change gets and hopefully the quicker the remainder learn new habits.

So I’m left to ponder, just how often have we really changed things? How often have we fooled ourselves into believing we’ve achieved a revolution when actually we’ve managed little more than a slight evolution. If everyone can understand something so quickly is that not because it fits with existing understanding and behaviours? If everyone gets to carry over their creature comforts (gets what they are used to) have we not re-asserted old world thinking?

Technorati Tags: business, philosophy, technology

That’s right I’m stupid, I must be cos I learn something new and significant every day.

And I’m not talking about some API detail or some new framework I mean I’m changing my thinking, I’m changing me, I’m seeing a bigger picture. I want topsight^[1], not nitty gritty detail. Detail is easy to cope with if you can see the higher view, how things fit and interact.

If I were clever, there’d be nothing more to learn and I’d have answers to all questions. Knowledge is all very fine but it comes through learning and that comes through experiencing things and that’s what makes you smarter. Having a big brain isn’t enough.

[1] Topsight - a term used by Gelernter in MirrorWorlds, meaning the ability to view the whole system rather than small details.

Urgh, they’re threatening to deploy the Land Warrior System in spite of the reservations of the men in the firing line.

The problems with Land Warrior are seemingly obvious:

1. Having every soldier trackable individually is simply going to overload the command and control system. We have our armies broken down into units under a hierarchical structure for a reason which is to strike a balance between overload and oversight.
2. Being able to peer around a corner without exposing oneself (because of the “clever” camera on your gun) is all very fine but consider this: your enemy is smart enough to hide until you and your buddies do expose yourselves allowing for a more effective ambush.
3. Advanced signalling breaks - hand signals are by far the best way of directing people under these extreme conditions.
4. Soldiers won’t have time to gaze at their heads up display to check on where everyone in their unit is positioned before taking action. Put simply, the enemy is not going to sit still whilst you plan out the perfect attack [”No battle plan survives contact with the enemy” - Moltke]

Perhaps more worrying is that the US military has had warnings about their excessive focus on technology previously.

This whole debacle is a classic example of what happens when those not at the sharp end invent things they think are a good idea and foist them on the hapless individuals they intend to “help”. It is a mistake repeated daily in IT projects around the world. For the enterprise this is merely serious, on the battlefield it’s quite possibly lethal.

Technorati Tags: military, technology

CCTV just got an “upgrade”

One more example of the utter inability of our society to grasp a fundamental issue - the law in all its forms including CCTV is merely a deterrent. You can scare away the amateurs from committing petty crimes like littering but it does nothing in the more serious cases:

“Yes ma’am I appreciate that your husband was stabbed to death in a horribly violent attack but don’t worry we’ve got it all on CCTV”

Too late! The attackers get thirty years at most whilst the husband loses maybe 50 years of his life and a wife and children suffer similarly. Justice is retrospective and it fails us in these moments. CCTV does nothing to deliver significant improvement because like so much else it doesn’t tackle the hard issues - why do we continually waste money on these largely pointless pieces of technology? Time to get real, please.

Technorati Tags: society, technology

Many a techie spends too much time looking inwardly at their favoured technologies without reference to what goes on elsewhere. This inhibits growth in a number of directions:

1. Frameworks don’t evolve significantly, we just polish them a little more
2. There are no genuine technology paradigm shifts
3. …..

Interesting then to watch how some of the “web crowd” are picking up on this. Can’t wait to see what happens next…….

Technorati Tags: web, technology evangelism learning

Update: A related post from Cote

Patrick has noticed the parallels as has Werner Vogels.

There’s an awful lot of stuff to be learnt about building distributed systems from the human body including how control isn’t centralized, how reliability is achieved and how the constant replacement of cells contributes to overall health.

A couple of books worth a read:

• Imitation of Life: How Biology is inspiring computing
• Creation: Life and How to Make It

Technorati Tags: distributed systems, biology, technology

A graphic demonstration of how short-term thinking over strategy can be outright life-threatening:

I was out driving this morning on a dual carriageway, approaching a slip-road in lane two (the outside lane) so as to avoid merging traffic. In front of me is another driver who in their impatience is on the boot-lid of another vehicle, we’re right on top of the merge point for the slip-road now which is a feeder from the M4. At this point, the boot-lid surfer dives into the inside lane clearly intent on an undertake, failing to account for the motor-cycllst merging from the slip road. What follows is best described as a lucky escape with both motor-cyclist and boot-lid surfer taking evasive action.

Many a technical decision is taken without measuring, pausing for strategic thought or evaluating alternatives. What we typically do is jump on whatever is the current hype without bothering to weigh up whether it’s worth it. This is a key contributory factor to the utter absurdity that is the hype curve where we all get far too optimistic about a technology, jump on it and then after a few years realise it is not the holy grail. Techies as a group are supposed to be some of the most rational, analytical of all and yet we make this newbie mistake - who are we kidding?

Short-term thinking can be seen elsewhere too such as in the feature’s versus refactoring/re-structuring discussions that go on in software product (shrink-wrap, web or otherwise) companies across the world. In general it seems that businesses encourage these minimal thought policies. Increase customers now. Increase revenue now. Don’t worry about consequences we’ll deal with those later………maybe!

But wait a minute - if a child wants to stick their fingers in the wall-socket should we just forget the consequences and let them do it? Clearly not - that’s strategic, big picture thinking which saved our child some pain. Why don’t we do similar things and save our businesses, systems and workers pain?

I do wonder if rampant short-termism is somewhat related to our own survival instincts. Perhaps when there’s too much pressure/threat we are more likely to stop thinking about the big picture? Understandable when it’s life or death but surely unnecessary in business, tech or the daily commute?

Technorati Tags: business, strategy, technology

I attended Sun’s London Tech Days conference yesterday. I came away with a couple of overwhelming impressions.

We have serious problems in terms of the quality of our developers. Some of the questions being asked should be reserved for those getting to know the trade at school or university and not be uttered by “paid professionals”. Simply horrifying……and here’s some further food for thought.

Sun have this saying “The Network is The Computer” - based on yesterday I’d say that’s true for all the wrong reasons. See what they really mean is “The Network is One Computer”. There was lots of talk about multiple cores, multiple threads (no mention of the likes of SEDA), race condition detectors (a cool piece of tech by the way), optimizing compilers and so on.

But there wasn’t a single mention of what they would do when you grew beyond a single machine. The closest they came was a few brief mentionings of clustering which only counts for tightly-coupled, uniform computational scenarios. All those web 2.0 sites, Amazon, Google and so on are building systems on top of multiple boxes which need to work together in ad-hoc, changing configurations.

In our current systems construction doctrine we still focus on building our application inside of a single machine out of bits (e.g. Spring or App Server style). Witness how we strive to allow developers to run all that’s required on their own machine and rarely force them to run remotely. Do we really feel this is a good thing given that when the code is given to ops the first thing they do is put it on lots of machines? What incentive does a developer have to write monitoring tools to help ops out when they don’t see the pain in development?

Standard development practices are not aligned with what goes on in deployment, Sun and others need to start bridging that gap and developers need to start working in a style that fits much better with what goes on “over the wall”. Certainly it’s not going to be easy given we have such a long legacy of confining ourselves to a single machine but it’s necessary.

The fact of the matter is we know developers don’t like threads, transactional memory might be a nicer model but it won’t scale any better and virtualization makes it easier to spread software across multiple “machines”. The future looks like it will be multiple, co-operating, separate (possibly remote) processes - now where are the tools etc to make that happen?

Technorati Tags: distributed systems, engineering, sun, technology

Dave has some thoughts on the recent W3C “Web Of Services For Enterprise Computing” get-together.

He notes:

“Given all the angst about Web architecture vs Web services architecture, I was also surprised that there was no support for technical reconciliation. I suggested WADL (Web application description language), to help with enterprises and the desperate perl/python hacker building stronger typed REST services. And for the flipside, I suggested improved SOAP to URI/XML bindings so SOAP/WSDL services would be more easily consumable by REST clients.”

No support for technical reconciliation? Why would either side want technical reconciliation when each believes it’s approach is “right” for it’s own needs?

I think he hits the nail with a comment earlier in the posting:

“There was almost no discussion about how enterprise computing is different from “non-enterprise computing””

Which is sad but not surprising - a good amount of the reason we have the whole Web/WS-* debacle is down to the fact that this kind of discussion didn’t happen when it should have. And if we still aren’t having the discussion, technical reconciliation is likely to be impeded.

Technorati Tags: distributed systems, enterprise, web

Apparently not Anne:

“…..But I’m not so good at blending into the traditional, primarily because I’m just not interested in doing that.

A further issue is that though I have the background to comment on enterprise software, it doesn’t excite me anymore. And every minute I spend looking at enterprise software is a minute I’m not looking at the latest in social media and social tools and new social forms on the web, my real area of interest…..”

For all that enterprise is big business and pays the bills, in many cases it’s not all that interesting. Once you’ve done a couple of integration jobs, you’ve done them all.

All of which leaves me wondering:

1. Might there be some connection between repeated reinventions of the wheel and the run-of-the-mill sameness of most enterprise jobs?
2. How much talent can one really expect to attract into enterprise?

Technorati Tags: enterprise, technology, web

I’m not talking personal hygiene which of course is important (your Mum is right, brush your teeth, bathe regularly etc) I’m talking protocols.

Pete’s highlighting the issue which comes down to this strange comment from Paul.

SOAP isn’t RPC? Well sure if I read this I can see the statement Paul references:

“SOAP is fundamentally a stateless, one-way message exchange paradigm, but applications can create more complex interaction patterns (e.g., request/response, request/multiple responses, etc.) by combining such one-way exchanges with features provided by an underlying protocol and/or application-specific information.”

Trouble is the same document also says:

“[SOAP Part2] defines a data model for SOAP, a particular encoding scheme for data types which may be used for conveying remote procedure calls (RPC), as well as one concrete realization of the underlying protocol binding framework defined in [SOAP Part1].”

And then (here):

“One of the design goals of SOAP Version 1.2 is to encapsulate remote procedure call functionality using the extensibility and flexibility of XML. SOAP Part 2 section 4 has defined a uniform representation for RPC invocations and responses carried in SOAP messages.”

What does [SOAP Part2] say?

“One of the design goals of SOAP is to facilitate the exchange of messages that map conveniently to definitions and invocations of method and procedure calls in commonly used programming languages. For that purpose, this section defines a uniform representation of remote procedure call (RPC) requests and responses.”

If we go by the letter of the base-spec SOAP isn’t about RPC but if we look at the spirit, it’s stated that a design goal for SOAP is to support RPC. Note the number of times RPC is mentioned in the base and adjunct specs. Seems like RPC is a fairly fundamental part of this whole thing. Confused?

Update: Don Box wrote up this brief history of SOAP which also makes the link to RPC

Technorati Tags: distributed systems, SOAP, technology, web