Pragmatic Dictator

Any time we make a decision to ignore a problem, or attempt to work around it, we make more work for ourselves.

That additional work turns into something akin to the death of a thousand cuts. The problem never goes away and the associated consequences continue to manifest all over the place. And as the problem gets tougher so there are more and nastier consequences to deal with and they are harder to work around. Eventually, something has to give…….

This is really a long hard-learned life lesson which has some convenient applications in software engineering…..

Consider a bug in some code you’re using - if you don’t fix that bug, you end up dotting workarounds all over to cope with the presence of this bug. The same problem occurs with a badly written API, all around your code there will be compensations for it.

I was reminded of this recently when discussing code-lifecycle issues which is a nasty problem, make no mistake. Some members of the discussion wanted to accept the issues as unsolvable and work around them leading to hideously complex architectures and masses of code scattered all over the system. Myself and a couple of others felt it better to tackle the problem head on and came up with a simple solution that’s small and easy to use. Most importantly, the solution relieves us of the burden of tackling all those hideous consequences.

This lesson can also be applied to systems architecture in general. For example, the general policy to place firewalls at the borders of corporate networks which accept nothing unless it’s on port 80 has resulted in a raft issues which exert significant costs on our software and hardware infrastructure.

Technorati Tags: design, philosophy

Is a (popular?) saying that essentially means “There’s more than one way to do something”.

And the fact that there are many ways to do things makes for a lot of fun, after all variety is the spice of life! But it also brings a burden of having to sort through the options and pick the one most appropriate.

This should all be horribly familiar to most of us but I’m going to concentrate on engineers for the rest of this blog. See, engineers do some peculiar things when confronted with this reality:

1. They invent all sorts of complex classification schemes in an attempt to determine the best solution.
2. They often ignore available real-world data or worse the fact that there is no data.
3. They declare one solution the solution for now and all time ignoring the fact that things change.
4. They attempt to force their choice of solution on everyone else - “my way is the only way and all should follow because that’s what I’ve decided is best for you”.
5. Once they’ve decided on the solution, no question or opposing view is acceptable.

If you have any of these tendencies take note:

1. You are not infallible, your solution may not be the best.
2. If someone else picked a better solution, it could cost you big time.
3. If you keep pushing your solution without listening to others, you risk making yourself an outcast.

Then take a break, go get a chill pill and mend your ways. Be a little less prescriptive and thrive on that variety.

Technorati Tags: philosophy, engineering

Steve Jobs had some bad stuff to say about Java and Jonathan Schwarz has responded as part of his most recent interview with Scoble. If you haven’t heard any of this before, get the scoop here.

Now, whether or not you agree with Jobs on his stance in respect of Java and the iPhone, he is right on one thing: Java is getting heavyweight. And it’s not because of the language or even the platform but the way in which it’s all delivered to the desktop, server etc.

It comes as a monolithic package, all or nothing. You can extend it via classpath or whatever as you wish but there’s something deeply wrong with this picture……

And that is the fact that Java is designed to be dynamically extensible not monolithic and static as is the prevailing pattern pursued by Sun’s JDK development team and the world of J2EE. Remember, Java in it’s early days was all about code-downloading and dynamic extension. Yet, all of those leading the Java masses seem to insist on ignoring this fact and continuing to build structures that are perhaps even less dynamic than C++ with shared libraries?

Worse, we’ve tried to fix this sort of thing by adding support for scripting languages etc “because Java isn’t dynamic”. And we’ve got all these static compile time constructs we’ve added when perhaps a better solution would be better support at runtime to trap and fix these problems.

We’ve been dropping the ball for a long time - we’ve ignored a core construct of Java and now we’re complaining about the consequences. To Schwarz, Sun and the Java masses, stop leading Java down the path of static, ever-larger, monolithic bloatware. Go back to Java’s core, fix it and start building again. If you need help with understanding what might need doing, go read this paper then go talk to OSGi and the Jini community.

I posted recently on the increasing interest in making IT more green. But keeping control of power consumption and it’s associated side-effects has been with us for some time. Google have been facing this issue and published articles and papers on it some time back. Now, one could argue that Google are an extreme example. But think about this……….

It doesn’t matter how big of a data-centre you have or need, you face the same issues Google does. And those issues cost a lot of money and given that enterprises are concerned with cost reduction, it surely makes sense to pay more attention to this subject.

So far we’ve only talked about the hardware, what about the software? We all know that whilst XML is “more open” (I’d debate even that argument) it costs a lot of CPU time, we also know that due to network infrastructure preferences we do a lot of un-necessary polling which also costs CPU time. And we compound this by what we encourage in modern programmers who’ve never programmed in assembler for a system with maybe 4k of memory and a 1Mhz CPU and thus haven’t learnt to be very careful about resource consumption. Those same programmers are not encouraged to focus on the quality of algorithm selection or design, code is king.

If we are to tackle the power consumption issues, it’s not just the hardware that needs to improve, we need better architecture which can dynamically adapt to platform and load changes (Amazon EC2 anyone?) and a new generation of tools and frameworks but, just as important we need programmers who will raise their head out of the code and view the consequences of their actions from a more global perspective. Business needs to look long, avoiding the rampant short-termism that is a key signature of recent times and realize that focusing harder on design rather than just cranking out code will save them money over time because their systems will be more modular, more efficient and less costly to maintain.

Finally if that’s not a compelling business case, consider that recent history shows the consuming public is starting to demand that car manufacturers etc produce greener alternatives and a well known hardware manufacturer is leveraging this in their marketing. It’s entirely possible this trend will spread to websites where consumers will choose your competitor because they can say “we are 50% more power efficient”.

Technorati Tags: distributed systems, amazon, enterprise, green, ec2

Inspired by this post I wanted to list a couple of things I (and a few others) have been thinking about.

What’s the big cost of all that XML we’re using? Lots of parsing with associated CPU thrashing, all over the place and there’s all those custom hardware-based accelerators.

Then there’s all the endless polling we do for news feeds or because we have to get through firewalls or we’re doing AJAX or whatever.

I also wonder about the tradeoff between all these heavyweight software stacks we have to “make programming simpler” and the amount of hardware it consumes to run that stuff. Might it be better to make the programmer think a little harder and focus more heavily on efficiency?

Much of what we’re building eats through excessive amounts of CPU, memory, disk and network leading to greater power consumption. The CPU suppliers have been thinking about it for a while and, as usual it seems the software crew are taking a long time to catch up.

Does anyone else see some irony in the fact that one of the supposed leaders in this area also employs “Mr XML”?

If we’re really interested in being green, it’s not enough to buy green hardware, you have to watch how you use it.

In recent discussions here and elsewhere, it’s become apparent to me that people aren’t aware of some fundamentals of how Jini lookup behaves and some utilities one might typically use to avoid rather obvious pitfalls.

When one performs a successful lookup, a proxy is returned. Because the proxy is a serialized Java construct we may need to download some code (if the proxy is a JERI reflective proxy, there’s no code to download for example). For the sake of this discussion, let’s assume we do need to pull some code down.

What happens under the covers, is we check to see if we already have a classloader for the codebase associated with the proxy. If we do, there’s no need to download the code, we already have what we need. Otherwise we must instantiate a new PreferredClassLoader and download the necessary bytecode.

And for as long as we have a reference to the proxy, the classloader will exist and thus any further code-downloading will be driven by the usual Java rules for classloading.

Obviously if we release our references to the proxy, it’s classes are no longer referenced and the classloader might well be gc’d as it should be.

Potentially, that means that everytime we do a lookup/use proxy/release cycle we could end up having to do code-downloading. Fortunately, there’s a utility available which avoids that issue and, as a side benefit keep’s it’s list of services up-to-date - it’s called LookupCache and can be created via ServiceDiscoveryManager.

Rather than do a direct lookup, we construct a LookupCache with the details of the kind of service we’re interested in and leave it to populate itself appropriately which includes handling the code-downloading aspect. All we then have to do is query the cache. And because the service proxies are now cached we are assured of not needing to repeatedly free and re-instantiate classloaders and incur the overhead of un-necessary code-downloading.

Some reasoning about REST - right or wrong:

1. REST is an architectural style
2. Basically then REST is a collection of architectural patterns I apply to an application
3. And an application built on REST is The Web

Thus is The Web not just an application? Does it not have a set of core libraries built around a set of core standards that express a set of patterns defined by REST? And in a lot of cases, we’d like to reuse those core bits of code and standards. But remember that all these things, whilst guided by REST principles are application specific that is specific to the application known as The Web.

And we know from experience with reuse elsewhere that sometimes we can only reuse patterns because the implementations of those patterns cannot be reused for our application.

Does it not then follow that for certain applications, reusing The Web application (http etc) is inappropriate but reusing REST patterns (caching etc) might be appropriate?

Kathy Sierra nails it.

This issue of consensus and everyone’s opinion having equal weight and deserving equal attention pops up all the time in development. Experience counts for a lot and we’re not real good at recognizing that these days the result is we make stupid mistakes over and over because newbies are given too much latitude and the old dogs are “so yesterday man”. Of course, there are projects that get stuck in a rut precisely because the old dogs do need to learn some new tricks.

Consensus can really only be achieved in a very small group with similar mindsets and even then it can take a lot of arguments and therefore a lot of time. I don’t think it’s a surprise then that many of the “successes” (be that Linux or Apple) are run by benevolent dictators as that seems to me to be the only way to make reasonable progress.

It will be interesting to see how this all plays out in respect of Apache River….

There are so many arguments concerning the merits of one technology over another. And sadly most of these arguments are pointless because the perspective governing the arguments is unreasonably biased towards one or other technology.

For example, when we compare JavaSpaces with JMS, we do so from the perspective of what we’re used to with JMS. This means we consider a JavaSpaces-based equivalent using all the old patterns, approaches etc of JMS. Then we look at the results and pronounce that, in the best case, JavaSpaces is no better than JMS. How could it be better? You effectively constrained your non-JMS based technology to being used in a JMS based way.

Essentially, we frequently are engaging in what are apples and oranges comparisons believing they are apples and apples.

There are a number of side effects to this behaviour:

• It inhibits the rate of progress in our industry because we continue to design the same way and demand the same old features regardless of their suitability.
• By treating everything the same, all discussion of advantages is reduced to ‘ilities such as performance as these are the only remaining visible differentiators.
• The designs we use are rarely as good as they could be.

In general, to measure the true suitability of a technology to a problem one must solve the problem in the appropriate way for each technology we are considering rather than from the single standpoint of a technology we are already familiar with.

[Inspired in part by some bits of this posting and associated comments.]

Blitz has taken something of a back seat for a few months whilst I’ve been running around consulting, doing talks/presentations, evangelizing and taking a well-earned holiday.

Over the last couple of months I’ve been working with eTapestry who have some JavaSpaces stuff in their production environment. Amongst other things, we worked through some performance and architectural stuff but a key piece of feedback came in the form of a test which was showing up some problems in Blitz’s caching/indexing code.

I’ve been working on those issues for a while now and it’s lead to a significant simplification of Blitz’s core architecture with some useful performance benefits and a reduction in code-size.

I’ve also added some new features one of which is the ability to run user-specified code at boot-time. The user-code receives a copy of the Blitz proxy just prior to it being published via JoinManager into Jini lookup services. This neatly solves a common set of problems associated with initializing content in a JavaSpace and some other interesting issues. There’s also a much broader range of statistics available via programatic access covering number of dirty entries in cache etc.

I’ve upgraded to the latest version of Db/Java after working closely with Sleepycat on a bug found in a previous release which caused me to do a release of Blitz back-ported to a previous stable version.

At this point, I’m stabilizing the codebase, doing some profiling and some soak tests including the usual thrash tests and memory-leak detection routines. If things go well, I should have a new version out early in January.

Update: And it’s out - on New Years Day and in case you’re wondering, that’s just a happy accident, if I’d attempted to do it deliberately you’d still be waiting! It’s on SourceForge in the usual place and you should consider it alpha for a while - I have thrashed it but it pays to be paranoid…..

Still suffering from the marketing issues from way back when. Sadly, many a Jini evangelist still has vestigial aspects of the old marketing in their explanations. Time to clear the decks - here’s my Jini in a nutshell for techs:

• It’s about software services - doesn’t matter whether they run on servers, desktops, laptops, phones, toasters or anything else.
• Just because some piece of hardware uses Jini doesn’t mean that it must embed the entire Jini stack, support multicast etc.
• Jini services don’t have to be all Java.
• Services at compile time, know what other services they need to function - and it’s specified in a fashion that an IDE can reverse out for you (if I just had time to write the plugin).
• Services at compile time don’t need access to (via classpath etc) the stubs or proxies of the services they are dependent upon.
• Services can at runtime find the other services they need (without need for static naming strategies) at which point they receive the proxy or stub required to communicate with/use that service.
• The way in which services find each other is such that there’s no risk of the usual dependency loops where service y needs service x before it will start and vice versa.
• Jini Service deployment can be done in completely non-stop fashion - no restart of containers, web servers etc.

Once you’ve got your head around the above, you should probably get to grips with the Jini System Architecture which breaks down into three categories:

• Programming Model
• Infrastructure
• Services

All of which are covered in the overview section of the architecture document. Read the rest of the document and you’ll hopefully see that Jini systems aren’t like other kinds of system and require some new modes of thought.

Inspired by this from Fuzzy.

As we all know, there's an ongoing debate across mailing lists and the blogosphere about REST versus WS-*.

One of the key areas of discussion comes down to the assertion that only the web-technologies (as favoured by REST) can provide internet scale solutions. But does this really stand up?

The web is when all distilled a pile of cross-linked, navigable information with a standard set of operations. The thing is that these standard operations whilst apparently simple aren't simple come implementation time. This is because these simple operations against resources actually map down to a whole series of other operations. For example, let's say my Servlet based web-server receives a POST of some description. Some servlet must go away and perform the relevant associated operations, tweak a database, write a message to a queue etc to get some work done.

Thus, somewhere, at some point, your systems will have to do some hard work and that work must be made to scale. That is to say, you must for example, have scalable, transactional data-storage.

So then, Enterprises don't just need web-scale, they need web-site scale. That is to say, they must scale their internal systems to support the load they are placed under at the web-site.

Thus the question remains - how do we scale our services? And as demonstrated by the likes of Google and Amazon, the answer is not about REST it's about architecting systems that support horizontal scaling. Just check out this recent slide set from Dan Pritchett of eBay which, incidentally uses J2EE etc (it's also worth paying attention to how they scale their databases).

So am I anti REST? No, but I'm not pro either. REST has some admirable traits such as simplicity etc but those properties are generic targets we should all be striving for in our designs. And I think it's safe to say that few of those generic design centres have been a focus for WS-*. But for me, the real problem with the whole WS-* effort is that we haven't got enough distilled experience to:

1. Determine what standards if any, need creating.
2. Determine what the real problems and the optimal solutions might be so that we can standardize.

Experience comes first, then standards follow - not for the first time, our industry has tried to do it the other way around.

In summary, when comparing REST and WS-*, I don't think the web-scale argument really stands up as a justification for choosing one way or the other. But from a purist perspective, one can't help but notice that one of them seems to hit a lot more of the “right” design buttons than the other.

Tired of your job? Need to hire developers? Visit DZone Jobs: great people, great opportunities.

There is a saying “Don't flip the bozo bit” - I have a new one:

“Don't flip the Dalek bit”

That is to say, just at that point where you wish to “Exterminate!” the developer who broke the build, you should go get a cup of camomile tea and think happy thoughts for 30 minutes.

Tired of your job? Need to hire developers? Visit DZone Jobs: great people, great opportunities.

As noted in my previous blog entry, I talked at the inaugural Next Net meeting last night about the work I’ve been doing for Lone Crusader on top of Amazon’s EC2 and S3.

Thanks to all those who turned out and asked some great questions, including James Strachan (glad you liked my S3 locking hack!), Matthias Radestock from LShift, Adewale Oshineye from ThoughtWorks and the BetFair gang.

BetFair should also get a mention for supplying the beer and pizza.

Big thanks to Nigel Warren (NetCaboodle/Brunel University) and Ewan Silver (BetFair) for putting in all the organizational effort to get this started - looking forward to the next meeting already guys.

Finally, for those that requested a copy of the slides, you can find them here.

Technorati Tags: Jini, EC2, S3, Amazon, Next Net

Tired of your job? Need to hire developers? Visit DZone Jobs: great people, great opportunities.

I'll be speaking at the inaugural meeting of “Next Net”, a forum for the discussion of distributed systems issues such as deployment, tools, management, utility computing etc.

My presentation will cover my thoughts on utility computing and provide details of my work with Amazon's EC2 and S3 service offerings.

Date for the meeting is 4th December, 6pm. You can find more details and signup here.

Tired of your job? Need to hire developers? Visit DZone Jobs: great people, great opportunities.