Still Can't Pin the Tail on the Donkey
Posted by Dan Creswell in Distributed Systems, TechnologyI attended Sun’s London Tech Days conference yesterday. I came away with a couple of overwhelming impressions.
We have serious problems in terms of the quality of our developers. Some of the questions being asked should be reserved for those getting to know the trade at school or university and not be uttered by “paid professionals”. Simply horrifying……and here’s some further food for thought.
Sun have this saying “The Network is The Computer” – based on yesterday I’d say that’s true for all the wrong reasons. See what they really mean is “The Network is One Computer”. There was lots of talk about multiple cores, multiple threads (no mention of the likes of SEDA), race condition detectors (a cool piece of tech by the way), optimizing compilers and so on.
But there wasn’t a single mention of what they would do when you grew beyond a single machine. The closest they came was a few brief mentionings of clustering which only counts for tightly-coupled, uniform computational scenarios. All those web 2.0 sites, Amazon, Google and so on are building systems on top of multiple boxes which need to work together in ad-hoc, changing configurations.
In our current systems construction doctrine we still focus on building our application inside of a single machine out of bits (e.g. Spring or App Server style). Witness how we strive to allow developers to run all that’s required on their own machine and rarely force them to run remotely. Do we really feel this is a good thing given that when the code is given to ops the first thing they do is put it on lots of machines? What incentive does a developer have to write monitoring tools to help ops out when they don’t see the pain in development?
Standard development practices are not aligned with what goes on in deployment, Sun and others need to start bridging that gap and developers need to start working in a style that fits much better with what goes on “over the wall”. Certainly it’s not going to be easy given we have such a long legacy of confining ourselves to a single machine but it’s necessary.
The fact of the matter is we know developers don’t like threads, transactional memory might be a nicer model but it won’t scale any better and virtualization makes it easier to spread software across multiple “machines”. The future looks like it will be multiple, co-operating, separate (possibly remote) processes – now where are the tools etc to make that happen?
Technorati Tags: distributed systems, engineering, sun, technology
Entries (RSS)
Couldn’t agree more Dan. First, any solution/algorithm is thought of with a single-machine (=> shared memory) in mind. To make matters worse, even for such (relatively simple) scenarios, rarely is any thought spared for functioning in the face of (complete/partial) failures.
“Developers don’t like threads” I have to disagree Dan, developers LOVE threads, everyone thinks they are the absolute master of threads, unfortunately 99% of those people are completely and utterly wrong. If the buggers didn’t like threads they wouldn’t try and write multi-threaded code to “optimise” their application into a death spiral.
The average level of skill in IT is lower now than it was 10 years ago and the drop in formalism and computer science is even greater. Then again…. Consolidated.java…..
I’d be cautious about generalising about the overall quality of even the London Java development community from the crowd who attended Sun’s Tech Days. It may not have been a representative sample.
I was in the building next door attending QCon. The quality of discussion was very high, at least in the Banking Architectures track that I attended, and people were focussed on the issues in building large scale distributed systems using tools like Javaspaces, grid computing, guaranteed low-latency JVMs and implementations of the AMQ Protocol.
When you consider your new role at Betfairl you really should have been at QCon. I think you would have enjoyed it more.
Hmm…Maybe we should talk to Ewan about getting the AMQP guys to demo their implementations at a NextNet evening?
No sample is ever representative
To be fair I shouldn’t have attributed my comment about developer quality just to that particular event. It’s something of a trend I’ve seen across a number of events and business sectors in recent times.
I certainly saw value in Qcon for me but not enough to justify the cost involved remembering that I’d be paying for myself cos I don’t start with Betfair until next week (disclosure: Floyd and I had discussions about discounts but at the time I couldn’t make a solid commitment due to calendar fog, them’s the breaks). I’d have loved to have caught up with Werner Vogels not sure however that I would’ve wanted to do the discussions in the banking track. I’m sure that sounds a little odd until I mention that I have a lot of beer and back-channel chat with many of the people involved in the banking stuff.
In respect of Betfair, well it’s a different beast from the banking arena which is one of the reasons I’m interested in it. It’s worth saying that at least some of the tech you mention above isn’t stuff I would consider appropriate for Betfair’s problem-space.
Finally, Ewan, Nigel Warren and myself are “co-chairs” for NextNet so you can approach any of us about the AMQP stuff. Couple of things to say on this:
I’m probably going to setup something a bit more permanent for NextNet on Ning.
Ewan is pretty over-loaded right now so apply a load balancing policy when contacting “Next Net”
In respect of NextNet’s mission, it’s really about real-world experience, discussion of issues etc. I don’t think we want to have product pitches, benchmark discussions (mine’s bigger than yours!) etc. Demo’s then, could form part of some overall presentation perhaps on some cool messaging solution deployed in the real-world but I’d say demo’s should almost be an aside or something for beer and pizza time. Imagine a good NextNet meeting to be maybe like reading a good paper on swarm algorithms or utility computing infrastructure and challenges or quorum algorithms or parallels in biological and distributed systems but with interaction and discussion.