The vast majority of server-side Java programmers have J2EE on their resumes, they pride themselves for being experts in this particular technology but there’s a problem. Many of these programmers have their minds warped into the J2EE way of thinking:

1. There is nothing beyond the database
2. POJOs focused purely on business logic
3. This is distributed programming
4. Ops is someone elses problem
5. Deploy more or bigger boxes to scale

Most enterprises can comfortably tolerate systems built this way but what if you’re not most enterprises? What if you are an eBay or a MySpace? eBay for example have thrown out almost all of J2EE and built their own libraries to tackle the problems they face around:

1. Monitoring
2. Hot Upgrades
3. Scaling

Basically once you’re beyond a certain level of challenge the J2EE way of thought and patterns of design don’t work. So where does one find Java programmers that can cope with such a challenge? They’re going to need serious knowledge of:

1. Deployment
2. Monitoring
3. Networking
4. FLP
5. SEDA
6. Threads
7. REST
8. …..

But put that on a job advert and see how many responses you get! J2EE is a raging success to be sure but if you’re a company that can’t use it you’re likely going to be a victim of that success when looking to hire server-side Java programmers.

All of this has me wondering how one should frame job adverts of this nature. Should we even bother asking for Java experience or simply drop the language/platform constraint entirely? What should we be asking for? Multi-user online game programming perhaps? What else?

Technorati Tags: distributed systems, gaming, Java, J2EE, Web

Update: I’ve added REST to the list as I suspect that it won’t fit well with existing J2EE-derived thinking.

Update 2: For an idea of MySpace’s challenges see here and here.  And then grab a copy of the slides from the Mix06 site entitled “Running a Mega-Site on Microsoft Technologies” (under Breakout, Next Generation Browsing Experience).

And I quote:

“Project Darkstar is a research effort focused on the design of massive-scale, latency-optimized systems like online games. Written entirely in the Java programming language, the server platform provides a simple but powerful interface for defining server-side application logic. It takes care of persistence, load balancing, consistency and communications, leaving developers free to focus on their applications.

As of Fall 2006 we are in the final stages of a re-design and implementation of the system. Check back here soon for new details, programming interfaces, and news about public releases. ”

There’s some pretty serious people involved in that effort one of whom I was talking to over email earlier today, sounds like they’re doing some fun stuff.

Technorati Tags: distributed systems, gaming, sun

Manny is by his own admission a “pretty pictures guy” hence his interest in web clients, Ajax, REST etc.

His first posting covers the basics of the Betfair business model which will set the scene for future tech postings and discussion of the brutal challenges Betfair faces (expect to see some serious performance numbers).

Technorati Tags: betfair, blogger, technology

Well maybe - I certainly wouldn’t run Windows on a Thumper as they have over at Johns Hopkins but the tests are interesting for a couple of reasons:

1. They represent what might well have been one of Jim Gray’s last pieces of work.
2. There’s a headline figure of 9Ktps for an approximation of a large scale bank account processing system.

Technorati Tags: databases, performance, Benchmarking, technology

Urgh, they’re threatening to deploy the Land Warrior System in spite of the reservations of the men in the firing line.

The problems with Land Warrior are seemingly obvious:

1. Having every soldier trackable individually is simply going to overload the command and control system. We have our armies broken down into units under a hierarchical structure for a reason which is to strike a balance between overload and oversight.
2. Being able to peer around a corner without exposing oneself (because of the “clever” camera on your gun) is all very fine but consider this: your enemy is smart enough to hide until you and your buddies do expose yourselves allowing for a more effective ambush.
3. Advanced signalling breaks - hand signals are by far the best way of directing people under these extreme conditions.
4. Soldiers won’t have time to gaze at their heads up display to check on where everyone in their unit is positioned before taking action. Put simply, the enemy is not going to sit still whilst you plan out the perfect attack ["No battle plan survives contact with the enemy" - Moltke]

Perhaps more worrying is that the US military has had warnings about their excessive focus on technology previously.

This whole debacle is a classic example of what happens when those not at the sharp end invent things they think are a good idea and foist them on the hapless individuals they intend to “help”. It is a mistake repeated daily in IT projects around the world. For the enterprise this is merely serious, on the battlefield it’s quite possibly lethal.

Technorati Tags: military, technology

We rely a lot on hardware, tools or software optimizing automatically. We expect it from processors, compilers, databases, javaspaces, caches and so on.

But there’s a limit to what can be achieved because these elements can only make a best guess at what we are trying to achieve and optimize on that basis.

These guesses are based on limited context - e.g. an instruction stream or the recency and/or frequency with which something is updated. The big picture as to what’s really going on is in the surrounding application or maybe even just the programmers head.

Thus without programmer involvement, there’s always a limit to what we can deduce from limited contextual information. The problems come when our systems can’t deduce enough to be effective in their optimization. This can happen because there isn’t enough information (it may have been removed or was never present) or more interestingly because the programmer writes code that isn’t amenable to the optimization process.

Thus it is quite interesting that we spend much time hiding details like concurrency or distribution from our programmers (via frameworks or tools) actually preventing or at minimum discouraging them from getting involved in and specifying the necessary details to optimize effectively.

Technorati Tags: architecture, design, distributed systems, performance

…..what I’ve done

Linkin Park

Technorati Tags: music, linkin park,personal

CCTV just got an “upgrade”

One more example of the utter inability of our society to grasp a fundamental issue - the law in all its forms including CCTV is merely a deterrent. You can scare away the amateurs from committing petty crimes like littering but it does nothing in the more serious cases:

“Yes ma’am I appreciate that your husband was stabbed to death in a horribly violent attack but don’t worry we’ve got it all on CCTV”

Too late! The attackers get thirty years at most whilst the husband loses maybe 50 years of his life and a wife and children suffer similarly. Justice is retrospective and it fails us in these moments. CCTV does nothing to deliver significant improvement because like so much else it doesn’t tackle the hard issues - why do we continually waste money on these largely pointless pieces of technology? Time to get real, please.

Technorati Tags: society, technology

…..thanks for the memories.

Fall Out Boy

Technorati Tags: music,personal

Many a techie spends too much time looking inwardly at their favoured technologies without reference to what goes on elsewhere. This inhibits growth in a number of directions:

1. Frameworks don’t evolve significantly, we just polish them a little more
2. There are no genuine technology paradigm shifts
3. …..

Interesting then to watch how some of the “web crowd” are picking up on this. Can’t wait to see what happens next…….

Technorati Tags: web, technology evangelism learning

Update: A related post from Cote

That is the front page headline of The Sun newspaper with similar wording in The Mirror and in some of the local rags handed out around London. These headlines are of course related to the horrible business over at Virginia Tech and sadly, I believe they are making a bad situation worse.

These kinds of headlines demonize the attack and ultimately place the blame squarely on the gunman. Demonization has historically been used as a powerful tool for motivating armies to fight, just think about how a lot of people have described Iraqi forces or the Nazis. Fortunately the likes of Tim Collins have stood up and said this is not the right way and done something about it (meeting significant political resistance along the way).

So if the military have figured out that demonization is not a good tactic, why do we as a society allow this to be pushed through our newspapers and other channels? Perhaps because it allows us to blame someone else and move on? Make no mistake the incident at Virginia Tech is appalling but if we are to learn lessons we need to avoid blaming the gunman and start asking ourselves as a society some harsh questions like “why did we ignore the warning signs?”

Technorati Tags: Virginia Tech

There’s been a lot of chatter recently in the blogosphere about the technical direction of the database. This discussion has been ongoing for some time and dates back to at least Bosworth’s utterings.

It kind of reminds me of the old “we’ll never use that much memory” argument, “you’ll never need more than one database”. I wonder, have we got to the point where most decent size systems be they web or enterprise will need to store more data than can be held in a single database?

Perhaps partitioning and use of dumber storage mechanisms¹ will become the norm? It strikes me that this might be a more promising approach² than attempting to build clusters when working with utility compute platforms like EC2/S3. Seems like we have some design patterns appearing, just need the frameworks to catch up?

[1] Might include RDBMS used purely for storage, containing no business logic.

[2] Partitioning might provide simpler and cheaper growth strategies than the aggravation of upgrading the server hardware for the database

Technorati Tags: architecture, storage

One evening an old Cherokee told his grandson about a battle that goes on inside people. He said, “My son, the battle is between two wolves inside us all..

One is Evil. It is anger, envy, jealousy, sorrow, regret, greed, arrogance, self-pity, guilt, resentment, inferiority, lies, false pride, superiority, and ego.

The other is Good. It is joy, peace, love, hope, serenity, humility, kindness, benevolence,empathy, generosity, truth, compassion and faith.”

The grand son thought about it for a minute and then asked his grandfather: “Which wolf wins?”

The old Cherokee simply replied, “The one you feed.”

[With thanks to Dave Zaffery for pointing me at this]

Technorati Tags: personal, philosophy

A transaction is an abstraction that provides some combination of the ACID properties.

This is just a concept. The trouble is we’ve created implementations of the concept that carry the same name. The result has been that we’ve forgotten about the concept and associate the term “transaction” typically with the RDBMS implementation of the concept.

This confusion can have significant impact when we get to design of a transactional system. What you really want to build is a system which provides the appropriate balance of ACID properties but what you usually end up doing is constructing your system around a database - i.e. we dumped design and went straight to a common implementation with no consideration for what we really set out to achieve.

And in a twist of irony, I was going to link to the ACID page over at Wikipedia but it makes the very mistake I’m blogging about! So instead I’ll point to Page 3 of Transaction Processing: Concepts and Techniques by Gray and Reuter.

Technorati Tags: architecture, design, distributed systems

It’s an all too familiar story, an overflowing inbox of rapid fire emails day after day. A ceaseless bombardment of meeting requests, news updates, design discussions, status reports, the list goes on. It doesn’t take long for this bombardment to push recipients into pavlovian responses such as:

1. If the subject doesn’t have immediate meaning, delete the email
2. Another status email which surely is like all the previous ones, delete it
3. Another request to a meeting - book it into the calendar without a second’s thought

These behaviours and others are quite damaging to your organization. Potentially useful information is ignored, people end up permanently in meetings, never getting anything done or pausing to think strategically.

So what’s the problem? To echo a previous post, email is a push mechanism and worse, it tends to be an interrupting push mechanism. We are all familiar with bouncing icons or fading dialogues displaying recipient, subject etc of a just-received email. Email can also be problematic to manage:

1. Mailing lists are difficult to keep up to date
2. Getting on and off mailing lists can often be more complex than is desirable
3. How do you find the right people to email to?

So what might we use instead? How about blogs and RSS feeds? Blogs are a fine place to publish announcements, news, status reports etc. We have a wide range of readers, online and offline and managing subscriptions is a key area of focus for these applications. RSS of course gives us a pull mechanism allowing subscribers to choose how often they receive the updates. All that’s left is to maintain a well known page to link the blogs (and you can regularly email a link to that page around) so that potentially interested readers can find the subject matter they require.

It all seems so obvious and yet there are still a mound of corporates out there that haven’t moved to this more balanced approach to information dissemination and co-ordination.

Technorati Tags: management, process