The thing about the average database is that it doesn’t really encourage similar behaviour. It is all too tempting (and easy) to just allow everyone to access everything. Whilst we confine ourselves to a single application using the database, the problem is to some extent contained but often what we actually do is allow multiple applications access to the same database. The exact way in which this is done varies:

1. Sometimes we bundle all our middle tier code together even though it has separate roles and responsibilities and integrate all of it via a single database.
2. Sometimes we have multiple applications each running in a different process.

With each application we put on top of the database the problem gets worse increasing the number of invisible dependencies tying unrelated elements of code together by virtue of accessing a shared schema.

What’s happening is we’re sharing too much intimate knowledge across our system, something we’re all taught to fear. The solution is as always to prevent direct access to this intimate knowledge by interposing layers of abstraction. One way to do this is by requiring access to data to be wrapped up behind an interface. Historically we’ve done this by having a system own the database and expose interfaces that other systems can use to get the data.

Unfortunately there is a well-known issue with this approach which is that the level of granularity is wrong and these additional integration interfaces rapidly balloon into complex beasts. What we need is a a database wrapping entity that has a finer level of granularity than an entire system. Then the integration interfaces will be simpler because there will naturally be a less complex schema underpinning this more limited functionality.

What are we talking about? Services. We end up with a system of lots of discrete services each wrapping up their own data storage.

There are other benefits to this approach:

1. Each service can utilize the most appropriate storage option for it’s contained data whilst having zero impact on other services that might have different needs.
2. Each service is an independent entity that can be managed (monitored, deployed etc) separately.
3. Centralized access patterns are more easily broken down which is useful in cases where we deploy across multiple data-centres.

Who would do such a thing?

Technorati Tags: architecture, database, distributed systems, enterprise

1. They represent what might well have been one of Jim Gray‘s last pieces of work.
2. There’s a headline figure of 9Ktps for an approximation of a large scale bank account processing system.

Technorati Tags: databases, performance, Benchmarking, technology

“Like most architectural choices, the stateless constraint reflects a design trade-off. The disadvantage is that it may decrease network performance by increasing the repetitive data (per-interaction overhead) sent in a series of requests, since that data cannot be left on the server in a shared context. In addition, placing the application state on the client-side reduces the server’s control over consistent application behavior, since the application becomes dependent on the correct implementation of semantics across multiple client versions.”

Thus while statelessness is often claimed to achieve scalability, in certain applications that may not be the case due to the resultant load on the network.

Our pursuit of statelessness leads us to behaviours such as making a single entity responsible for the maintenance of all state. Often it’s a database that becomes a black hole sucking up hardware, network bandwidth, admin time and endless tuning effort. It also becomes the focus of our reliability concerns, with a need for clustering, RAID arrays etc. Stand around long enough and you’ll hear terrified utterings from staff such as “if we ever lose the database….”

Making some single thing responsible for all these aspects of our system is asking for trouble. Having all these heavyweight concerns squeezing down on a single element ultimately leads to breakage.

History shows that we aren’t entirely happy with this “single point of responsibility for all state”. We have cookies in browsers, local storage in browsers, thin clients that rely on servers to store all state and so on.

Perhaps we’re ignoring an underlying message: Maintenance of state is a shared responsibility for a system. We should seek to place that responsibility in appropriate places at appropriate times and be much more aware of responsibility boundaries and when it’s appropriate, share that responsibility amongst components.

Generally we consider TCP to be responsible for ensuring that state makes it to the other end of the connection. One hands some data to the TCP layer and we expect that it will ensure the data reaches the recipient. But is this true? What happens if we suffer a power outage before TCP transmits the data? When the machine restarts, is TCP going to restart and resend all that unsent data? Clearly not, whoever delegated responsibility to TCP for this data will now need to take steps to recover the situation.

What about a message queue? Typically we place some data in the queue and demand that it absolutely must deliver that message and not lose it in the meantime. That’s an awful lot of responsibility for a single component to carry! As an aside we’re also potentially making that queue a performance bottleneck of the future.

Then there’s the Web which in many cases puts responsibility on the client for maintenance of state. This is achieved through retries, restoring backups, re-entering details etc. Notably, this is the case even if the client “fails” e.g. your home router goes down or the PC overheats. There’s a certain amount of illusion here too where we believe the responsibility for state maintenance has been placed elsewhere e.g. Flickr. Ideally they don’t want to lose all your precious pictures but if they do, who will have to restore all that information?

I think it’s interesting that placing such responsibility with some single entity is perceived as the easy solution but it has a lot of hidden costs like redundant hardware, clustering, strict data-centre environment control, backups etc.

Spreading responsibility might ultimately be easier and fit with our desire for utility computing but it’s not commonplace and thus we’re lacking well documented patterns, software components etc. We are seeing some examples however, I would speculate that S3‘s API is the way it is precisely because it relies on spreading responsibility for state across a co-operative shared-nothing system rather than placing it all in a single shared-everything cluster.

Technorati Tags: amazon, distributed systems, enterprise, web, utility computing

“W3C members have issues surrounding enterprise computing. Not just distributed computing, but the typical concerns around transactions, scalability, high availability, and so on. Not only that they’re also dealing with 15, 20, and 25 year old technology that works just fine but is getting more and more difficult to maintain and doesn’t play well with others. Finally, there’s the issue of interconnecting these systems to each other, to partners, and to the Web.”

Once 15+ year old technology is getting difficult to maintain and doesn’t play nice with others, surely it’s time to considering throwing it out? Surely keeping it just forces all the new stuff we do to get bogged down and stifled supporting the old stuff?

I’m just wondering if the logical destination for this will be we can’t make progress any more because we get completely bogged down in our legacy? I certainly appreciate the companies holding on to so much legacy consider it important that the issue is tackled but should the rest of the world which is perhaps more flexible have to care? Should we constantly be bending all the good new stuff we do to cope with all this old crud?

If you buy a car there’s a limit to it’s useful life. You can keep running it after that but maintenance bills get higher and higher to the point where one gives up and just buys a new car because it’s cheaper.

Note also that the owner of this “legacy” car is the one suffering the high maintenance cost, not the manufacturer or the mechanic. Admittedly, the mechanic and the manufacturer must hold onto tools and parts but they can at a point of their choosing drop support. Should it not also be the case that companies holding onto legacy systems suffer similar high maintenance costs? Should these companies be able to displace those costs onto others in the form of horrendous complexity and endless legacy considerations in new products, specs etc?

Clearly there is no right answer for this dilemma but it might be time for us all to sit down and consider the whole cost of holding onto legacy which goes way beyond the simple increasing cost of maintenance.

If nothing else it strikes me as interesting that whilst IT pursues the concept of endless life for systems, nature provides many examples of limited lifespan as a useful tool for progress and sustainability.

Technorati Tags: enterprise, technology

Enterprises certainly want the money saving opportunities of utility computing options such as EC2 but there is evidence to suggest that RDBMS’en aren’t cut out for this role.

And recent statements on TSS suggest some people are at least beginning to think that we are mis-using RDBMS’en ignoring it’s lack of suitability to a task because it’s an easy option.

Time to seek a new religion?

Technorati Tags: amazon, distributed systems, enterprise, databases

“Unfortunately, Web Services, at least the WS-* style, are “Web” in name only. While WS-* enables tunneling over HTTP (used merely as an XML message transport), in almost every important aspect, WS-* violates (or at best ignores) the architectural principles of the Web as described in the W3C’s Architecture of the World Wide Web, Volume One and in Tim Berners-Lee’s personal design notes.”

Personally I find this statement completely unsurprising as all it really says is that The Web is not the same is WS-*. Perhaps some people are upset because WS-* aka Web Services dares to make use of the word Web. That might even be a reasonable complaint but does it really matter? It seems to me that this part of the paper is a lot more incendiary and far more significant:

“The large set of WS-* specifications is almost entirely focused on recreating traditional middleware capabilities using XML as the syntax for the formal message structure and the formal interface description.”

Which appears to be saying (at least to me) that WS-* is nothing new and thus all the hype about how it’s a great step forward starts to look rather lame. That’s not to say WS-* won’t have it’s uses but it’s no more of a cure-all than The Web or anything else.

[Update: Bill has clarified the posting I link to above, stating that his interest is in the fact that the likes of Gartner are acknowledging that The Web does indeed have an architecture.]

Technorati Tags: architecture, distributed systems, technology

There are other problems too like the fact that clustering in this fashion is infrastructure specific not application specific. Basically, the infrastructure will offer you configuration options that fit with what the infrastructure can support/implement generically in an application unaware fashion. You must then fit your application around what the infrastructure will offer which can mean design or performance compromises or maybe maintenance load (hand-holding by system admins etc).

There can be no argument that many adopt this approach and accept the compromises and for a certain class of system, as I’ve said, it’s probably acceptable. However, the typical profile for most systems is that they grow and evolve over time such that those compromises you made are no longer acceptable or viable. They become an albatross around the architects neck. Witness the number of people entrenched in battles to get scaling out of application servers or to add new services etc.

This approach to clustering often goes hand in hand with the denial of the realities of networking. It’s often assumed nothing breaks, there’s always enough bandwidth etc. And everything is coded like it runs in one JVM creating a single difficult to manage monolithic piece of code (something I find ironic given that the people that adopt this approach quite often talk about loose-coupling, service oriented architecture, networked applications etc) which is deliberately naive of it’s under-pinnings.

In an ironic twist, the people that choose to build things this way then throw their hands up in horror and complain when things become difficult to scale, or they haven’t got the level of control they want or they don’t like the way their code has turned out or whatever. What did they expect? They turned all these responsibilities over to some vendor’s software stating “I don’t want to deal with that, you handle it, just tell me what to do” and then in the very next breath said “why on earth did you tell me to do it that way?”

So if I don’t want to suffer this, what might I do? Start looking at building more modular, network aware modules which can be dynamically interlinked at runtime. And find a way to achieve resilience/scale at application level using simple unreliable components. I said in my previous posting:

“I can’t help but feel that there must be a better way……..”

Yep, there is – and I have a prototype to prove it. And, in case you’re wondering, yes the prototype does have persistent state and no it doesn’t require RAID or any other cool hardware, a bunch of blades on a network is all that’s needed. And of course, it uses Jini.

Whilst I understand the technical reasons for having these API’s – basically that a “managed environment” can’t support the existing equivalent API’s in the JDK (java.util.Timer and java.util.concurrent.*) there are some other things about this work that I find a little concerning.

Firstly, that part of the justification is about having simpler API’s than what’s in the JDK. What does that say about what’s in the JDK? Which programmers are the intended audience for the JDK API’s versus those in the JSRs which are J2EE oriented. And what does that say about the respective levels of ability of these audiences? Are we really suggesting that J2EE programmers aren’t as smart as those that might use the JDK? That doesn’t sound like a good thing to me for all sorts of reasons.

Secondly, these JSR’s are to all intents and purposes replicating API’s we already have in the standard JDK. Were I to see this kind of duplication in a customer’s architecture I’d be seriously concerned and quoting such things as the DRY principle (which actually goes way beyond the simple issue of not duplicating code). Is it really a good thing for J2EE to be going down a path of duplicating existing standardized functionality bent to it’s own needs? Might there not be something wrong with the model?

I gotta say that these JSR’s have a bit of an odour to them – good or bad I haven’t decided yet……….