Pragmatic Dictator

For a number of applications, the standard JavaSpace interface’s single Entry operations are sufficient to construct a scalable and simple solution.

However, there are certain cases where perhaps we’d like to write or take a large batch of Entry’s. With the existing interface this is possible but would incur a significant number of network round trips which can hurt performance. Fortunately, with the release of JINI 2.1 we gained an additional interface JavaSpace05 which provides bulk write/take and some other goodies (which I’ll mention later).

Not only do these bulk operations permit transfer of multiple Entry’s but they also have a provision for the use of multiple templates using an OR-based matching strategy. i.e. take an entry if it matches template A or template B or template C. As with the original operations you get timeouts, leasing and transactions.

Other additions include a second notify method (registerForAvailabilityEvent) which provides more detail concerning the lifetime of an Entry. For example, one can now receive an event to indicate the availability of an Entry as the result of an aborted take.

Lastly we have contents which allows for a form of multiple read against multiple templates. This is treated slightly differently from multiple take in recognition of it’s non-destructive nature. Some developers like to view this feature as an iterator but there are some key differences such as the ability for iteration to never end (due to a constant stream of new writes) and remoteness.

Currently Outrigger (the example JavaSpaces implementation in the Jini Starter Kit) and Blitz provide complete implementations of JavaSpace05. GigaSpaces provide their own equivalents but as yet haven’t announced compatible support for this new standard extension.

11/01/06 – Updated to include more commentary on tradeoffs and reliability

3/01/06 – Updated to include ifExists issues

Plenty of people have asked me to add clustering to Blitz and I’ve certainly been spending time looking at that idea but it’s time for a confession:

I dislike clustering intensely!

Especially when I consider it in the context of JINI philosophy. See, clustering is an attempt to hide and handle partial failure seamlessly allowing clients to imagine that all is well. It offends me on other levels as well – clustering is not simple to implement, not simple to deploy and not simple to manage. Not exactly a great example of the KISS principle and it flys in the face of the “Recovery Oriented Computing” approach taken by the likes of Google and Amazon.

Here are some of the other things that cause me to pause for thought:

(1) Data partitioning – how does one partition Entry’s across multiple JavaSpaces? There has to be some key field that we differentiate on. Or we just differentiate on Entry type which means all Entry’s of one type will end up in one place – not good for scaling. Okay, so which field is key? How do we express that? Do you want it in the Entry itself or in a configuration file? Do you want to change that dynamically or shut everything down, reconfigure and restart. And make no mistake moving data around as the result of changing configuration is going to be slow and painful!

(2) Load balancing – If one uses a master-slave approach, there’s no need for load balancing, there’s only one machine. But if you’re using multiple active machines you need to select a collection of potential query nodes on the basis of the partitioning information above after which you can consider load at each node to determine a choice. But how do you measure that load? Queries per second? Network traffic? CPU time? Disk load? And there’s another nasty factor that kicks around in this mix………..

(3) Replication – if you have multiple active machines serving queries for the same set of Entry’s, each machine has to inform others of changes to state via something like two-phase commit, paxos or whatever. So more nodes sharing Entry’s means more network traffic as they arbitrate over state changes. i.e. Just because you have several nodes providing access to shared data doesn’t mean you’ll scale because they must arbitrate in the same way as SMP processor boxes must. This multiple active machines approach works really well for read-mostly loads but JavaSpaces are update-mostly because take and write are the common operations.

(4) Performance through asynchronous replication – you can’t do this and be an official JavaSpace implementation because part of the JavaSpaces specification (under Operation Ordering) says:

Operations on a space are unordered. The only view of operation order can be a thread’s view of the order of the operations it performs. A view of inter-thread order can be imposed only by cooperating threads that use an application-specific protocol to prevent two or more operations being in progress at a single time on a single JavaSpaces service. Such means are outside the purview of this specification.

For example, given two threads T and U, if T performs a write operation and U performs a read with a template that would match the written entry, the read may not find the written entry even if the write returns before the read. Only if T and U cooperate to ensure that the write returns before the read commences would the read be ensured the opportunity to find the entry written by T (although it still might not do so because of an intervening take from a third entity).

Pay particular attention to that statement about co-operating threads and write/take. Note that when the write completes, it must be visible to the take immediately. Were we to use asynchronous replication, we would not be compliant with this requirement. I suspect given the way one should use take this shouldn’t matter in practice but it has implications for timeliness which might be inconvenient.

(5) Network partitioning – to handle these problems between client and cluster-member requires multiple network routes into the cluster. One cannot select another cluster-member if they are all accessed through a single network pipe. The same principles apply to intra-cluster communications. Whilst there are algorithms to tolerate these problems the ability to make progress under updates will be inhibited and clients may be repeatedly attempting to act on out-of-date state only to be aborted at transaction commit time as the algorithm determines that there is no viable resolution at this time. And even once the network is fixed, there will be a period of instability as things come back into sync.

(6) ifExists – fundamentally, the ifExists variants peer inside of transactions. In a master slave configuration this is not an issue but when one has multiple active partitions, there is a requirement to co-ordinate actions across all nodes which may have a possible match and a blocking transaction. Worse is that you can’t optimize this in any fashion, when the ifExists is issued, one must query all relevant bits of the cluster, arrange for any necessary callbacks and block accordingly.

Whilst it is possible to tackle all of these issues in a clustered implementation, a developer would only have so much control and it mightn’t be what is required for a particular application. There are all sorts of issues which may occur that have strange or undesirable manifestations from an application perspective and are not easily cured. The benefits of such a system are dubious in my opinion given the typical operation profile of a JavaSpaces application.

I can’t help but feel that there must be a better way……..

Other Observations

Many talk about a need for clustering when what they actually need is reliability. That is to say they want their systems to continue to make progress in the face of failures. There are many implementation options available for achieving this and clustering is but one. These same people then make things worse by also placing the requirements for load balancing, partitioning etc onto the cluster solution. This makes things considerably more complex by inter-mingling all sorts of complex and conflicting requirements. This results in either a clustering solution that is horribly difficult to configure or a less configurable solution that has a fixed set of trade-offs assumed for these inter-mingled requirements.

“Continuing to make progress in the face of failures” is a very foggy statement from an engineering perspective. Do we make progress across all work regardless of number of failures. Do we mean make progress with most work in the face a fixed number of failures or some other variation? If you want progress for all work regardless of number of failures, you better have big pockets for a lot of kit. Further, your performance ceiling will be limited because solutions for this problem require a lot of network traffic between members of the partition responsible for the data. More members means more resilience but more traffic and therefore less performance.

Still working away at this one. Blitz now has an integrated experimental version of my nio based transport which can be configured on or off. It currently only boosts, write, take and read performance (roughly 100% faster than normal). I’ve not tested it hard enough to be confident of heavyweight concurrent loads and once you start mixing in transactions, you only get maybe 20% performance boost. I figure I’ll have to extend the transport to support the TransactionParticipant interface as well.

I’ve had some chats with Peter Jones over at SUN and he’s given me some notes and some pointers as to where he sees issues in the current JERI code. He’s not been able to do much in that direction owing to commitments to JDK RMI which is a shame (IMHO) as JERI is way better and ought to be getting the attention. Ah well, maybe one day.

Anyway, the plan is still to go digging through JERI and find the problem – I have a few hunches but I need to run some more tests. My preference is definitely to fix up JERI and then junk the experimental transport (which won’t ever provide much in the way of security etc unless someone else wants to do that work).

Recently, Blitz has been getting some commercial interest which has kept me busy especially when one of my customers hit a deadlock. I’ve written that same customer a remote bootstrapping application that allows them to remotely load their entire JINI application and get it up and running. The real killer is that each downloader can be given a different profile as required by the system admin – this has many uses including testing new builds, value add for specific customers etc.

In between that work, I’ve been developing a JavaSpaces/JINI application rather than an implementation. It makes a nice change to work above the JavaSpaces API rather than under the covers. It allows me to spend time looking at Blitz from a users’ perspective and tweaking things accordingly. Of course, I’m rather an advanced user so it’s only so useful. Thus far, it’s meant a couple of additional configuration variables and a few fixes for the experimental nio transport.

So what is the application? Sorry, it’s remaining cloaked for now. What I will say is that it’s not going to be a pure JavaSpaces/JINI play – you can certainly access it that way but it’s likely going to have a REST-based web interface as well. There’s lots of things you could do with it but I have some specific ideas which will shape it’s direction for a while (no, I’m not telling). At least initially, it won’t be opensource which is something of a departure from the norm, instead it’ll be provided via Lone Crusader in some manner. I’ve already developed a couple of new patterns, not sure how to/if I should make those public. Bit of an experiment, should be interesting……

All the details are at SourceForge

If there are no last minute complaints, this release will become 1.13 (once I’ve updated the documentation) and become the first fully supported JINI 2.1 release.