Pragmatic Dictator

Some of the more common software development mistakes I’ve seen…..

Ignoring the triangle - The triangle represents a trade-off between three core elements of software delivery - resource, product (features, non-functionals, quality) and schedule. One can only ever control two elements, the third being determined by the decisions regarding the other two. So if one wishes to dictate product and schedule, sufficient resource must be made available to complete the task in the allotted time. If one wishes to dictate product and resource, then the schedule cannot be limited. It is simply “as long as it takes”. And if one wishes to dictate resource and schedule, then product features, quality etc must be traded away to allow completion of development within the time allotted.

It’s amazing how often organisations attempt to dictate all three elements and are then surprised when a project gets messy. Of course, development processes have evolved in recognition of this trade-off - agile for example is great for prioritising, dropping features and getting something useful out the door in a resonable timeframe with limited resources.

Heroic efforts - these are a bad sign. A regular pattern of projects turning into mad hack-fests, saved by some apparently super-talented individual(s) is indicative of broken processes. One step in addressing this problem involves an honest surgery immediately after the project to determine root causes (e.g. inadequate risk management) of the meltdown and methods of prevention for future projects (e.g. regular risk review and identification of appropriate mitigations).

In the very worst cases, management actively encourages such heroism via recognition and reward. Worshipping this kind of carnage and supposed miracle recovery is tantamount to approving bad project management. Note that well-intentioned management can unknowingly drive this behaviour. From McConnell’s Rapid Development:

Some managers encourage heroic behaviour when they focus too strongly on can-do attitudes. By elevating can-do attitudes above accurate and sometimes gloomy status reporting, such project managers undercut their ability to take corrective action. They don’t even know they need to take corrective action until the damage is done. As Tom DeMarco says, can-do attitudes escalate minor setbacks into true disasters.

No Risk Management - One can never predict or spot all the risks but there are some obvious ones that get missed over and over. For example, we’re building a piece of software that relies on a component we’ve not used before. This is a big risk, one that can be mitigated by writing a test-harness or simulation of the way in which we plan to use the component.

The simulation should include realistic load, failure conditions, maintenance etc and should be as close to the beginning of the project as possible to surface any issues early (we cannot afford to wait until final QA or deployment testing). There can be no shirking here because should our chosen component fail, we will need these tests in place so we can validate potential replacements as quickly and easily as possible.

Waterfall Agile - We’re supposedly “doing” agile but one or more of the following are true:

1. There’s a fixed deadline, with fixed features and fixed resources.
2. All Negotiation/Trade-off is done prior to project commencement with no review between sprints.
3. All sprints have been planned out in advance right up to the release date with no spare time.
4. There are no risks to manage (because there aren’t any apparently).
5. No-one is entertaining the idea of unknowns.
6. When a sprint doesn’t deliver as anticipated, outstanding work is simply crammed into the remaining sprints.

A couple of false economies software development indulges in:

1. It’s quicker for me to write the code than explain the design to someone else.
2. Automated deployment will have to wait until we have more time.

Number one costs a software development team in a number of ways:

• The career development of other members of the team is slowed - if one never discusses design how does one expect to obtain good designers or architects?
• The team’s development capacity is reduced - essentially projects bottleneck around the uncommunicative heroic individual.
• The team’s effectiveness is reduced - project load cannot be divided efficiently because individuals have skills in narrow areas limiting the breadth of work they can perform.
• Team morale is damaged with other developers feeling left out, unfulfilled and unable to influence project decisions

Number two yields costs including:

• We save on some development time but the cost is re-surfacing in staff-hours required to perform the deployment.
• An increasing number of mistakes that extend deployment time or breaks releases.
• We save development time once and pay the price for that saved time with each and every deployment.
• The cost of each successive deployment increases because the system’s size is growing.
• As each deployment takes ever longer, the gap between releases is likely to increase.

Recently I’ve had reason to go back over some data-structure theory, average times for insertion, deletion and search etc. I was reminded of a period around 1999 when I was implementing a Hyperchromatic Tree, a subtype of the basic red-black tree.

Hyperchromatic trees are lazily balanced i.e. balanced in the background rather than as part of the operation that changes the structure of the tree and they have some challenging locking disciplines to implement as well. They are designed to perform better under concurrent access than their ancestors.

Looking at my notes I was running a single test performing concurrent operations continuously for periods of 5 days against an instance containing 1 million entries on JDK 1.4. I suspect this period of my developer life taught me more about deadlocks and race conditions than any other.

Technorati Tags: concurrency, development, software

This just shouldn’t be a surprise. Frankly it’s closing the door long after the horse has bolted - check out Martin Odersky’s comments in this article from way back when. Who wants to bet that in spite of all this we still get closures in Java?

We make this mistake all the time with languages, frameworks, applications and operating systems. When will we learn the lesson I wonder?

Technorati Tags: philosophy, development, software

Let’s say you sell pottery. Your core business is about making that pottery, perhaps to order but there’s a whole load of additional work you must do around accounting and the like. Perhaps you also ship your pottery via surface carrier, this introduces further work tracking customer details (watch our for pesky privacy regulations), working with couriers etc.

Almost from the get go these days you’ll automate as much of this drudgery as possible using computers, essentially creating a one person IT department that supports your business processes. Obviously the bigger the business gets the more automation you will require, perhaps developing custom software to support the execution of your business processes (all that drudge you have to do but isn’t core). Your IT department will have to grow accordingly, hiring developers, project managers etc.

This standard tale of the relationship between business and technology plays out in many an enterprise every day. IT and in particular software development is a necessary evil, an undesirable cost to be carefully managed.

However for a certain class of enterprise, software is the product. Maybe it’s a shrink wrap product or a set of services provided to customers via the web. There will still be a need to have an IT department to support the business processes and it might well develop software of it’s own. There’s a fundamental difference between these two types of software because one of them is the business’ money generator. This software must be nurtured and tended to carefully, one can’t just add features, one must care about stability, performance, availability, scalability and so on. Money put into this effort is not a cost it’s an investment. Fundamentally, this software is not enterprise IT as most know it.

This key difference is oft-missed by management, vendors and analysts leading to confusion and unnecessary mistakes. I’ll cover one such mistake (related to SOA) in a follow-up post.

Technorati Tags: business, engineering, software

Planning and estimation discussions always come back to:

1. Agreeing what will be done
2. Agreeing how much it will cost
3. Agreeing a deadline by which the what will be done

Because these questions require that one knows everything down to the deepest detail and that all possible happenings are known (which means knowing when people will be ill and for how long, how much time they’ll need to take off for dealing with family troubles, problems with the plumbing etc) and the risks are mitigated such that they absolutely will not affect your project in unpredictable fashions.

That’s not to say that one can’t set deadlines but one has to expect to trade features away, adjust resourcing etc. Of course none of this is news and yet so many places claim to be agile whilst continuing to have the what, how much, when discussion.

Technorati Tags: development, engineering, process, software

That’s what everyone craves. That’s why lines of code is still seen as a useful measure in various quarters. That’s why we obsess over IDE’s and other so-called productivity tools.

Trouble is every decent software engineer knows that actually you want less code. It’s easier to maintain, easier to change, easier to debug, easier to build on. They’ll also tell you that the best way to build big codebases is out of lots of small, well isolated, loosely coupled bits with minimal knowledge of each other.

The less code philosophy requires doing some design, pausing for thought and not cranking code. It can provide massive benefit but it’s difficult to measure in any simplistic fashion and thus is seen as pure cost by many.

More code is the hare, less code is the tortoise - know which one I’d bet on.

Technorati Tags: development, software, systems

For a long time, software development within many an enterprise is treated as a subservient entity. Something that is expected to comply with the demands of the business without complaint and with limited options for pushback.

I believe the software we produce should be viewed as a stakeholder in it’s own right. It has it’s own needs for survival and ongoing growth and if these are always placed behind everyone else’s (i.e. the business) considerations, the results will be a long slow, painful death where the software becomes more and more brittle, less and less maintainable and staff productivity drifts down as staff turnover creeps up.

Consider a car - it has needs, new tyres, oil flushes, new exhausts, paint chip removal, new springs, new clutch etc. Fail to address those needs and your car will turn into a pile of rust before your eyes with all the attendant issues of depreciation, lost investment, breakdowns etc.

Why do we refuse to accept that software has a need for the motoring equivalent of oil changes and the like? Probably because software is an invisible abstract thing such that only those working with it see the damage being done, problem cars are easier to spot for a greater percentage of the population. This isn’t really an excuse however because software gives warning signs just as cars do. If there’s steam coming from under the bonnet (hood) you’d go to a mechanic, if your software keeps crashing or upgrades keep failing or development takes longer and longer it surely follows that it’s time to visit with your software engineer.

[ You may have noticed I like car analogies - here’s another: You can’t endlessly tune a car, it will only go so fast. Any further increase in speed can only come from starting with a new car that has better basic performance ingredients. The same is true for software, eventually you need a redesign to make further progress because the initial assumptions you made have all been invalidated. ]

Technorati Tags: development, software, systems

I was reading this from Bill which follows on from Joe. And had a couple of thoughts:

1. Google seems to have applied N > 1 to everything, not just storage - that’s pure distributed thinking, not the norm for the majority of software heads.
2. Eventual consistency might be kinda like concurrent programming for most people - i.e. many like to program sequentially and with the certainty that x has been completed immediately, in order and within a known timespan. Concurrency, eventual consistency and friends aren’t terribly amenable to this programming approach and it consequently melts many a brain.
3. We need for a lot more people to understand CAP.

[Note for Bill should he read this: it seems like your comments are broken, I’m seeing server errors when I hit post]

Check out this job spec.

Notice anything interesting? It’s for a seriously heavyweight distributed systems engineer sure but look deeper. Do you see mention of a single piece of technology? J2EE? JavaScript? Ruby? No, right? How weird is that? How many job specs do you see like that? Surely what matters is whether you know JBoss or Websphere, Java or Erlang or Ruby?

What’s the deal? It’s recognition of the fact that building systems is about how you think and reason which requires sound understanding of theory and how to apply it. It doesn’t matter how much code you can write or in what language because delivering a project is about a whole lot more than code.

So often I see companies create job specs for engineers where the key requirement is to hire someone who can hit the deck coding like mad using whatever tools have been selected. To that end they load the specs up with endless tech hubris and at interview ask the details of this or that bit of syntax or API call. But what about the next project within the company where the tech is different? All those engineers that just got hired are now useless, they don’t have the skills and we lose time whilst they learn. Or we could fire them and hire another lot?

Of course what happens more often than not is that companies ensure they don’t use new tech. Instead they force new projects into using all the same stuff they used before. This is a design disaster as now technology is dictating not design or suitability to requirements. A company that follows the hit the deck coding mantra just has deathmarch and no career progression stamped all over it.

Keeping an eye on trends and keeping abreast of new technology is a good thing to do but the larger context of what to use when, when to build rather than buy, when to dump something because it’s warping the design, when to dump one design approach for another (e.g. going from centralized to distributed) and so on is what really matters. This requires thinking, not an encyclopedic knowledge of a huge number of technologies.

Tech is for sissies - Concepts, principles, patterns, measurement, theory and so on are what matters.

[Confession: The title for this entry was inspired by a recent piece from Pat Helland, one of my favourite thinkers]

Technorati Tags: design, philosophy, engineering, software, systems

This scenario get’s played out all the time in IT. The young guns claim that the old guys are out of it, don’t get the latest tech, aren’t smart enough whilst the old dogs smile and are heard to say they’ve seen it all before.

There’s a fundamental tradeoff at work here:

1. Intelligence allows us to at least potentially progress faster
2. Experience allows us to avoid making mistakes as we make progress

Thus a bright inexperienced person may make fast progress but they’re much more likely to make mistakes which will slow them down. In contrast the experienced person may make slower progress but fewer mistakes. Classic hare and tortoise. Who wins?

The nature of software is such that the mistakes we make can take a long time to manifest and when they do, they cost us big time. Thus:

1. Mistakes don’t result in short-term localized damage rather they are far more disruptive with long-term, difficult to clean up damage
2. The time between the root cause of the problem and it’s costly manifestation is large.

It follows that for our intelligence to count we must be able to see sufficiently far ahead to spot our mistakes coming before they get out of hand. Is this achievable? I think software history says it’s not and thus experience is our only tool for understanding root causes and spotting the early signs of an approaching asteroid.

I reckon there’s a lot to be said for the old tradition of master craftsmen handing down their knowledge and experience to apprentices…. (and perhaps the old dogs can learn a few new tricks along the way).

Technorati Tags: process, software, technology

Patrick on the lack of decent reasoning in WS-* world:

“It is precisely this lack of rigor that has failed ws-* from the start”

If this problem were confined to WS-* it’d be a cause for celebration…..sadly it’s pretty much everywhere in software these days.

Update: Was wading through my bloglines feeds and forgot about something Bill said a while back which is relevant.

Intel wades in on the “we can’t do any more magic concurrency for software” issue.

It’s been debated often enough and always seems to come down to the fact that the average programmer isn’t able to cope with concurrency and needs higher levels of abstraction to do it for them. The thing is, we already have such abstractions e.g. transactions and we know they can only take us so far. Worse there are other abstractions out there such as blackboard systems which these average programmers either can’t or won’t try to cope with.

So what is to be done? Well if the last couple of decades are anything to go by, absolutely nothing! Why? It’s the talent limit. How many chip designers are there in the world? How many motherboard designers? How many car designers? How many developers? I’m willing to bet that there are considerably more people in the developer category than any of the others. This is because we’ve lowered the bar in terms of developer quality to cope with the wide demand for bums on seats and lets face it, it’s unlikely that the average enterprise is going to change it’s policies in this respect. It’s interesting to note that it’s much harder to lower the bar in for example chip design, it either works or it doesn’t whilst software is almost expected to be flaky these days.

Until we decide to clean house in software land, we’ll not get progress on these thorny issues because they only matter to the few and when all said and done, there is a school of thought that might suggest that software is good enough, concurrency, efficiency, quality and green’ness be damned.

Update: An example of how challenging it can be to make concurrency easy can be found here. On the surface we’ve done some good things and yet we are still open to the simplest of errors (in this case a missing synchronized clause).