Victims Of J2EE Success

The vast majority of server-side Java programmers have J2EE on their resumes, they pride themselves for being experts in this particular technology but there’s a problem. Many of these programmers have their minds warped into the J2EE way of thinking:

  1. There is nothing beyond the database
  2. POJOs focused purely on business logic
  3. This is distributed programming
  4. Ops is someone elses problem
  5. Deploy more or bigger boxes to scale

Most enterprises can comfortably tolerate systems built this way but what if you’re not most enterprises? What if you are an eBay or a MySpace? eBay for example have thrown out almost all of J2EE and built their own libraries to tackle the problems they face around:

  1. Monitoring
  2. Hot Upgrades
  3. Scaling

Basically once you’re beyond a certain level of challenge the J2EE way of thought and patterns of design don’t work. So where does one find Java programmers that can cope with such a challenge? They’re going to need serious knowledge of:

  1. Deployment
  2. Monitoring
  3. Networking
  4. FLP
  5. SEDA
  6. Threads
  7. REST
  8. …..

But put that on a job advert and see how many responses you get! J2EE is a raging success to be sure but if you’re a company that can’t use it you’re likely going to be a victim of that success when looking to hire server-side Java programmers.

All of this has me wondering how one should frame job adverts of this nature. Should we even bother asking for Java experience or simply drop the language/platform constraint entirely? What should we be asking for? Multi-user online game programming perhaps? What else?

Technorati Tags: , , , ,

Update: I’ve added REST to the list as I suspect that it won’t fit well with existing J2EE-derived thinking.

Update 2: For an idea of MySpace’s challenges see here and here.  And then grab a copy of the slides from the Mix06 site entitled “Running a Mega-Site on Microsoft Technologies” (under Breakout, Next Generation Browsing Experience).

36 Comments

  1. “What should we be asking for?”

    Protocols.

    Apart their “why make it simple and fast when you can make it complicated and slow” attitude, most J2EE developpers know nothing about web protocols, something hardly surprising given the brain-dead share-everything-synchronous design of J2EE.

    Ask for HTTP/1.1, HTML 4.0, JSON and REST.

    And make sure to pick only developpers who know what Java is: “the simplicity of C++ combined with the performances of Smalltalk”.

  2. I think that you are absolutely right about diverging from J2EE if your requirements are unique.

    Nevertheless Monitoring, Hot Upgrades and Scaling are not that unique, I’ve never found one company that uses J2EE that is not frustrated to find out those requirements are not met. This can be seen as J2EE failure or as JEEv8 opportunity.

    The problem most companies confront is that if they don’t like J2EE they have to build their own in house ad-hoc solution. Probably eBay and MySpace have the $$$ to build their custom solution, but most companies prefer to play safe, and to play safe means going after a big company with a standard product (IBM, BEA, Oracle, etc.)

    Now if you look at those big companies with their standard products, you will notice that they bought those products, because the big companies have no way to build those products, they just can acquire small companies with competing products. The only exception to this rule would be BEA, I think they developed their product from the ground up and became a major player in the application server arena.

    All we can do is to build those projects in our garage and to wait for big companies to discover us.

  3. About 1. Monitoring:
    The situation is improving. But the basic problem from my point of view is, that adminstrators so far refused to learn about J2EE or have not been given appropriate resources to grok that J2EE “beast”.

    2. Hot upgrades
    can be done in J2EE. It’s a hard problem and it needs a lot of planning and investments up front. But project managers only want features done. Until the project is over and operations has all the problems like hot upgrades.

    3.Scaling
    J2EE supports this very well. But still you have to know what you are doing. Most developers still have to learn that the hard way.

    About Deployment:
    Because of the situation described in 1. we have no clear knowledge what “deployment” means in J2EE. There even a re roles defined for it. But in the end the developer had to fill them all. That’s what we need to change.

  4. Ade asked:

    “Dan,
    What’s FLP?”
    :) You’d have to come over for an interview to get that answer!

    More seriously, I’ll contact you off-blog with the answer in a little while. Unless of course a kind member of the audience wishes to wade in.

  5. Yet another unintended consequence of Sun pushing J2EE over Jini if you ask me. Everything was dumbed down and down and down. Most enterprises today are left with a clustered shared everything point-2-point-2-point mess terminating at 1 database that fails over in 2 scenarios out of 10 (the 2 never happen in practice). I wonder where we’d be if 10 years ago we started growing programmers the other way. We’d certainly have more stone killers. I have to imagine it would have been much better for Sun too – have they really made any money off of J2EE?

    I think that J2EE has been an absolute disaster.

    We are really in a crappy place right now all things considered on the server side. The good thing is ideas are brewing and a shift of some sort is obviously afoot. Conceptually it is there. Making it happen requires stone killers today. If we are lucky in 2-3 years there will be more available for the masses (including me).

  6. there are people out there using things other than big j2ee servers to build their systems. look for an emphasis on spring rather than jboss/weblogic/etc on resumes.

  7. Mike Herrick: “We are really in a crappy place right now all things considered on the server side.”

    You must have been locked up in boring company doing J2EE 1.3 projects for the last few years because really, the Java server side has never been so exciting. Look at all the great technologies out there. EJB3, Spring, many excellent (open source) frameworks to make your life simpler and develop more agile.

    Java is rockin’, you just have to open your mind and look further than just the stuff that comes out of Santa Clara.

  8. To address your question about finding qualified server side Java developers for your work. Make two posts. One that contains all your exacting requirements. One that fairly clearly states that it is for developing internal business applications in Java.

    Take the list of people of applied for the real posting and remove all the people who applied for the other one, because the people you want would be bored developing plain-old web applications. You’ll probably end up excluding some smart people who haven’t done the stuff you are looking for but could make the jump, but that’s the price you pay for easy filtering.

  9. I have been doing distributed programming for over ten years from BSD sockets, to Java, CORBA, EJB, JINI, Messaging, etc. I found J2EE containers were a great improvements over their predecessors. Though, I agree that J2EE umbrella means a lot of things to different people, and we should be using real words to describe enterprise level concerns such as deployment, monitoring, networking, concurrency, etc. For any large enterprise systems, the need for monitoring, scaling, security and availability are very critical. J2EE created a abstraction for developers for writing enterprise and distributed programming by implementing internal threading, transactions, security, remoting, and monitoring (JMX). Needless to say, it was marketed as a silver bullet by commercial vendors and abused by developers who didn’t understand the realities of underlying system. However, things like monitoring, hot deployment, security and high availability are really hard and I have not seen any reliable solutions outside commercial space. These problems should only be managed in a container or centralized way and are best handled at architecture level rather than at implementation level such as using J2EE. Though, they may be using some in-house frameworks for monitoring, deployment, etc. The scalability problems are solved by divide and conquer, i.e., define many services, which are managed independently. In most cases these services are stateless and store state in the database or some memory server. Such architecture can lend itself easily for high availability and can use any underlying technology. For example, at Amazon or Ebay, the site is backed by many many services and at any time, one or more service may be restarting, but it does not effect most of the users.

    As far as resumes are concerned, I understand most companies use keyword search for finding matching candidate, but it’s mistake. You can never rely on some computer program to present you the matching candidates. You will have to read the resumes and talk to the people to learn if they understand these concepts and have relevant experience.

  10. I think focus should not API of any language. We should always ask developers how things work behind the scene.

    for example most of people know what is application server. but if you any developer, what goes in making of application server or how can we start writing our own application server, there where maximum developer will fail, they most of developer, just focus on J2EE API and rarely looked beyond that and invested time in understanding how things work behind the scene or can we work alternatively without this xyz technology. maximum replies just like add more servers to handle load or stuff is also common reply when we ask about load. but we ask for fine tunning tips ?…very few knew it ….

  11. I think any blog (or blog responder for that matter) slamming J2EE at this point is just wildly out of date. The world has moved on; I doubt you can find a pure J2EE if you tried. I don’t know of any, I’m pretty entrenched in the Java web scene in my area. I think it’s pretty sad to techies randomly angry about things they don’t know about. Maybe you guys should go read another Paul Graham essay to chill out a bit.

  12. I started with J2EE in 1999 and within a year decided it was crap. There are many better ways of using Java in serving up web applications. However, try getting a job without 5 years of J2EE code monkey work on your resume. Delivering successful Java web applications without the usual JSP/Struts/EJB stereotype is a better sign of someone who can deliver using any technology.

  13. “how can we start writing our own application server”

    J2EE application server is not the bleeding edge of IT technology and I do not think any really good programmer will look into jboss sources just for fun, there are too many other interesting things to take care of in the spare time.

    In general J2EE is too big and too complicated and that is why they have ‘roles’ in the J2EE projects.

  14. As much as I hate to admit it Dan, I think you are right. Indeed J2EE made some collateral damage for it’s developers. I am one J2EE developer and I realized that if I keep swimming in these murky waters I might not get to my destination. I believe that once you have learned the technicalities in J2EE there’s not much challenge left and if you keep doing it you will limit yourself to a single way of thinking which is fine if you want to work with it for the rest of your life but if you like challenges and to continuously improve yourself you need to look around you and jump trains from time to time. Come to think of it, this is actually valid for any specific technology that is why I tried to invest lately in becoming a good programmer and software designer instead of learning all kind of technologies that appear on the market daily. Anyway, having a solid base in programming, I believe you can easily deal with any kind of specific technology.

  15. Ivan,

    “I think any blog….. slamming J2EE”…..

    I am not slamming J2EE, I’m pointing out that J2EE has corrupted minds into thinking a particular way. That particular mode of thought has then carried into other technologies (some of which are supposedly improvements over J2EE). Thus whether one considers J2EE out of date or not, the thinking is still alive and that’s actually what really matters because this or that technology is just a tool. Notice also that many of the things I list as being essential knowledge are design-level issues or higher.

    Finally note that the overall purpose of the post is about means for finding developers not corrupted by J2EE group-think.

  16. “Uh, Myspace is a .NET shop. And is a brilliant example of an exceptionally poor webapp.”

    I was using MySpace as an example of something that exhibits a set of characteristics that make J2EE a poor choice. I wasn’t intending that it be considered a technological poster child though I accept I could’ve phrased it better.

    Technology isn’t actually the issue either as I’m discussing underlying thought patterns. The trouble is that most developers don’t think in terms of patterns or architecture, they think in terms of technologies and allow that to shape their architectural approach. I’m interested in finding developers who think architecture and then technology.

  17. I think we should be optimistic, after all Sun has made great efforts to improve J2EE and JVM itself , i think EJB 3.0 is a success and we should give J2EE another chance.

  18. Hmm… I can say that I am J2EE developer. But I do not feel like having “corrupted mind” (well one can use this as an contrargument ;-)). I also know/use other things like Hybernate, Spring (that one I like very much), Struts, Eclipse RCP and Epinox and others. So it is difficult to say that I am somehow locked into “pure” J2EE. Perhaps my level is not high enough but I can not understand why adding servers to J2EE cluster is bad for scaling? Yes, performance issues are quit complex and additional servers not always help. But good j2ee developer is supposed to know that. I think the problem you mention is not J2EE specific. There are always developers with “shalow” knoladge of technologies and such people may “spoil” any good idea.
    You have also mentioned eBay as an example. I have recntly heard that they rejected using database transactions. It was said that eBay has a distributed database and distributed transactions were too slow. Instead eBay developers do things “properly” they update “important” tables first and in case of troubles they know how to clean them up. Well as J2EE developer I can hardly understend such explanation but I have seen developers who started rejecting db transactions using eBay as an example… ;-)

  19. Hi Yury,

    “Perhaps my level is not high enough but I can not understand why adding servers to J2EE cluster is bad for scaling?”

    If your load is read-only and you can afford to be loosely consistent then adding boxes is fine. Scenarios outside of this profile need other strategies (or at least a mixture).

    “I think the problem you mention is not J2EE specific.”

    Indeed however J2EE’s wide reach exaggerates any “bad” side effects such as “wrong-thinking” well beyond what other technologies might do.

    “Instead eBay developers do things “properly” they update “important” tables first and in case of troubles they know how to clean them up. Well as J2EE developer I can hardly understend such explanation”

    For a hint as to what’s going on here see:

    http://www.dancres.org/blitzblog/2007/04/12/when-is-a-transaction-not-a-transaction/

    And check out the eBay presentation:

    http://www.addsimplicity.com.nyud.net:8080/downloads/eBaySDForum2006-11-29.pdf

  20. Personally, I don’t see a problem with it. Most developers can spell J2EE, but understanding it is another thing all together. I encourage my developers to know various frameworks, whether J2EE or .Net or whatever. And I like to see various languages too. I don’t care about someone being the best C programmer or Java guy. I care whether he’s diverse in his tools and can pick the right tool for the right job and get things done quickly with as little heavy lifting as possible.

    I’ve learned a lot from J2EE. I can’t think of a reason I’d want to use it today. :) But I sure did a few years back. But isn’t that the case for just about every technology? You should keep pressing on, learn new techniques, languages, etc. I find that to be quite fun, challenging and rewarding.

  21. Mike Y:

    I agree with much of what you say but would contend that learning new technologies and languages is only the beginning. Understanding higher level issues around design, architecture, performance etc is where the real value is at though it should be underpinned with a grounding in various tech.

    At the moment I think too many are tech magpies that eschew the higher level thinking. And as I’ve said in several comments, the motivation for the entry is to determine how best to locate those that think at or aspire to be at this higher level.

  22. I think I’ve been a victim of J2EE success. It’s my own darned fault, but I see now where I got away from a more productive path.

    J2EE was supposed to solve all the hard problems for developers. The major ones. Transactions, security, remoting. That is fine. But for 95% of applications, this kind of architecture is overkill. It was sold as something you do so that you’ll be able to scale in the future. However, the fact that the biggest sites are moving away from J2EE at the super-high end tells me something: J2EE is not necessary at that scale.

    So J2EE is now not required at the high end, and a pain at the low end. Is there a sweet spot in the middle? I’m not sure.

    I’m learning Python and Django. I’m learning more about Unix. For too long I’ve just been staring at the Java JSRs and I didn’t understand that when doing maintenance, Operations and Reporting have to be easy.

  23. Oh, and to the guy again: Sorry I should have explained that F, L and P are the first letters of the authors’ surnames; (Michael) Fisher, (Nancy) Lynch and (Michael) Patterson. (Note not _David_ Patterson of operating systems book fame. Some people seem to confuse the two, I don’t know why).

    – D.

  24. Hi Steve,

    “I think I’ve been a victim of J2EE success”

    It sounds like you’re sorting out a cure though :)

    Seriously just remember it’s not just about the technology (though I think Python and Django are worth spending time with) it’s the thinking – your design philosophy, your architectural approach, taste, simplicity, pragmatism, a global view that covers the whole lifecycle including deployment, treating ops as customers etc.

    Good luck!

    Dan,

  25. To David Peterson,

    Spot on with the FLP thing – one nit pick which is that Patterson should be Paterson (at least according to all the history I can find) though there are mis-spellings around the web.

    So I owe you some beer tokens – I sometimes get to Sydney and If you’re ever in the UK I’m buying.

    Good stuff,

    Dan.

  26. The problem is that J2EE covers a way smaller niche than everybody seems to believe (probably due to marketing) that leaves the other niches “unattended” due to lack of hype :-)
    Even new frameworks appeared to cover the difference between what people think J2EE should do and what actually does, but still the focus is J2EE.

    As always, one must use the right tool for the problem…

  27. I’ve been looking for a developer for the past 4 months and really it is not easy to find a good one (i.e. knowing threading, protocols, persistence, scalability, object theory and other “basic” concepts).
    During my interviews of all of the senior developers, designers and architects I got one remark that was really representative of the state of the market today and the impact of the J2EE way of doing things. The remark was after I asked this “architect” about multithreaded programming, he said “I leave that for the people that know the stuff” refering that he used libraries or app servers instead of taking care of that part of the application.
    I agree that you shouldn’t reinvent the wheel, but if you are a car architect you should know why wheels are made that way and which wheel type to use. AND if you need to build a wheel, then at least you where to start.
    Nowadays market is full of people that apply recipes, not knowing if the recipe is appropriate for your needs. (If all you have is a hammer then everything looks like a nail).

    I almost forgot, apparently if you use Tomcat to put a couple of JSPs then you’re building a J2EE application and Tomcat is an application server. Back in the old days (1999) Tomcat was a servlet container.

    I know of a couple of aberrant architectures because of doing it in an app server (aka Tomcat). The application is a server that listens for SMS through a SMPP channel, that’s the main purpose, but since the application had a web service interface to query for transactions then someone decided to transform that into a web application!!! So now the server is embedded in Tomcat instead of embedding Jetty or Simple within the server!!!

    We develop gateways and systems that interconnect other systems, an app server is too much, too bloated for our needs, we need control, we need to know what’s going on in our application.

    Sorry for the long post, but it’s kind of comforting to see that there are people somewhere that think as I do and have the same problems as I do.

    If you find a way to attract real devs, please share it with us. I will do the same if I succeed.

  28. To Jorge -

    Agree with your sentiments. Just expressed something similar in email to Dan, part of which I will take the liberty of repeating below:

    My little secret advice for finding a good developer is to look for one who programs in at least one “interesting” language, where “interesting” is defined to be one of: Perl (decreasingly, but it was my interesting language for a long while so I’ll include it), Python (the new sweet spot it seems), Ruby, Haskell, Erlang, Concurrent Clean, Stackless Python, etc. This tells you (at least) two things about the person: (1) the person is familiar with (1) the idea of learning a language “for fun” – which in turn tells you a lot about their motivations to be in the industry, and (2) that they have been exposed to programming ideas outside of the conventional “narrow box” thinking of C-like (yes, Java) statically typed procedural languages.

    Now that doesn’t mean you’re expecting the person to be coding functors, monads, closures and higher order functions (“for food”, i.e. paid day-job) all day long, but take a bet that the kind of person who has been exposed to these concepts, *and* can also code a “day job language” like Java/J2EE is likely to be a lot better hire than someone who has not.

    I also know what you mean about the “arse-about application” anti-pattern. Because the “servlet container” is the canonical “core” of the environment in the eyes of the (inexperienced) architect, they go ahead and build the whole damn app inside the thing. The app gets started via a ContextLoaderListener hook or some similar bullshit, and if you want to refresh some aspect of the UI contained in a servlet or jsp (you do have run-time compilation switched off for performance, right? lol) then it’s a matter of taking the whole damn thing down to do so. Truly crazy stuff, but it comes with the territory it seems. Clearly a separation of concerns argument would have the UI tier and the processing tier in separate application contexts (and the processing tier not, ideally, running inside a J2EE app-server enviironment at all). I think the Spring guys, Adrian Colyer in particular, are starting to head this way with the talk of OSGI support and integration, but there is some way to go just yet.

    – Dave

  29. Okay last one (I promise!) – a rejoinder to Yury:

    Yury – you cite the example of eBay and their home-grown transaction management approach, but do you understand _why_ eBay made those technical decisions? Martin Fowler has a write-up on eBay’s “transactionless” approach (http://www.martinfowler.com/bliki/Transactionless.html) and Dan Pritchett’s InfoQ speaker page (with links to presentations) to which he refers is here: http://qcon.infoq.com/qcon/speakers/show_speaker.jsp?oid=175.

    You do state in your post that “you can hardly understand the explanation” so I hope the above gives you some information. Maybe it’s best described like that old country & western song “Freedom ain’t free” – except we can say instead that: “2PC ain’t free” ;) (2PC = two-phase commit) It isn’t, not for you, me or eBay. And when you’re running tens of thousands of database operations per second, the overhead of strict isolation and strong ACID compliance can begin to hurt!

    And while I’m on about 2PC, check Gregor’s article, originally entitled “Starbucks doesn’t use 2 phase commit”. Quite illuminating :)

    http://www.enterpriseintegrationpatterns.com/docs/IEEE_Software_Design_2PC.pdf

    In fact, Gregor’s whole site is worth a long read. He was one of the most impressive speakers (IMO) at SpringOne 2006 in Antwerp.

    Okay, stopping now – got _real_ (i.e. “for food”) work to do!

    Dave

  30. Pingback: Pragmatic Dictator » Blog Archive » Victims Of J2EE Success « A Frog in the Valley Internet Stream Pulse

  31. Pingback: Shahzad Bhatti » Blog Archive » J2EE Bashing

  32. Shahzad — Whilst I would agree with you that J2EE was an improvement over predecessor technologies for enterprise distributed programming (RPC and CORBA as perhaps the most obvious, but also COM/DCOM in the MS space as well as earlier middleware standards such as DCE also), I would take issue with your characterisation of Dan’s post as “J2EE bashing”. It is true that there has been a lot of anti-J2EE sentiment, especially since Spring/Hibernate started to come seriously onto the scene, but I don’t know why you would (seem to) feel that this is a bad thing.

    Frankly, the rise of Spring and Hibernate (and Toplink, and now JPA as part of the EJB3 spec) was really a grass-roots developer community response to extremely serious issues of sorely lacking functionality (AOP, DI etc) and serious performance limitations (EJB2) within the J2EE architectural stack. Spring and Hibernate were born from the painful experience of software developers and architects who built real systems and found that the (so-called) “J2EE Blueprints” just didn’t cut the mustard in a real-world scenario (especially a demanding transaction processing one).

    In fact, as a former CORBA developer yourself (as you state in your post), I would have thought you would have been well positioned to observe what is (in my view) the *fundamental* limitation of the early J2EE design model: its sufferance (to a large, and now thankfully a decreasing extent) from *design by committee* – the very same affliction that basically killed CORBA. The whole open-source “build first, standardise later” approach adopted by Spring / Hibernate, etc is the very antithesis of the former approach. Distributed innovation and “best technology wins” has finally started to trump “most powerful / influential vendor” and the results are (blissfully!) finally beginning to show.

    So I don’t think Dan was intentionally “J2EE bashing” in his post – but he was questioning the depth of knowledge held by the very many people now bandying “J2EE” (and related “techologies” e.g. JSP, JPA, JNDI, etc) around on their resumes. All distributed systems, Java or otherwise, are governed by the same underlying rules of computational theory. Many software developers would do well to read a good text on distributed systems (perhaps Birman’s latest book “Distributed Systems” or the slightly older one of the same name by van Steen and Tanenbaum) before getting too bogged down in reading vendor whitepapers or “specification” documents.

    Dave

Comments are closed.