• Consideration of many possible technology options
• Examination and identification of constraints
• Thought about the pros and cons of using various patterns and styles
• Comparing various splits of role and responsibility
• Looking at various tradeoffs of complexity versus function
• Formulation of opinion on possible future directions of system growth

A large proportion of this information is lost when the design document is written, because the focus is typically on providing a (notionally) definitive view of how a system should be structured which might be in the form of a bunch of UML diagrams or merely a collection of Visio-type diagrams and explanation of what each of the boxes in the diagrams does. At the code-level there is almost no chance that any of this information will have been retained. Yet this information is of high value since it:

• is the explanation as to why a design is the way it is
• provides reviewers with a clearer view of what was and was not considered
• forms the basis for assessment of the maturity of a designer and can be used for coaching/mentoring
• can provide insight for those with less experience
• contains assumptions which if breached by changes in circumstance would dictate a re-design
• dictates to a large extent how suitable for purpose a design might be

Thus I believe It’s important to expose elements of the act of design via documentation alongside the design itself, conversations during the design work etc.

Ignoring the triangle – The triangle represents a trade-off between three core elements of software delivery – resource, product (features, non-functionals, quality) and schedule. One can only ever control two elements, the third being determined by the decisions regarding the other two. So if one wishes to dictate product and schedule, sufficient resource must be made available to complete the task in the allotted time. If one wishes to dictate product and resource, then the schedule cannot be limited. It is simply “as long as it takes”. And if one wishes to dictate resource and schedule, then product features, quality etc must be traded away to allow completion of development within the time allotted.

It’s amazing how often organisations attempt to dictate all three elements and are then surprised when a project gets messy. Of course, development processes have evolved in recognition of this trade-off – agile for example is great for prioritising, dropping features and getting something useful out the door in a resonable timeframe with limited resources.

Heroic efforts – these are a bad sign. A regular pattern of projects turning into mad hack-fests, saved by some apparently super-talented individual(s) is indicative of broken processes. One step in addressing this problem involves an honest surgery immediately after the project to determine root causes (e.g. inadequate risk management) of the meltdown and methods of prevention for future projects (e.g. regular risk review and identification of appropriate mitigations).

In the very worst cases, management actively encourages such heroism via recognition and reward. Worshipping this kind of carnage and supposed miracle recovery is tantamount to approving bad project management. Note that well-intentioned management can unknowingly drive this behaviour. From McConnell’s Rapid Development:

Some managers encourage heroic behaviour when they focus too strongly on can-do attitudes. By elevating can-do attitudes above accurate and sometimes gloomy status reporting, such project managers undercut their ability to take corrective action. They don’t even know they need to take corrective action until the damage is done. As Tom DeMarco says, can-do attitudes escalate minor setbacks into true disasters.

No Risk Management – One can never predict or spot all the risks but there are some obvious ones that get missed over and over. For example, we’re building a piece of software that relies on a component we’ve not used before. This is a big risk, one that can be mitigated by writing a test-harness or simulation of the way in which we plan to use the component.

The simulation should include realistic load, failure conditions, maintenance etc and should be as close to the beginning of the project as possible to surface any issues early (we cannot afford to wait until final QA or deployment testing). There can be no shirking here because should our chosen component fail, we will need these tests in place so we can validate potential replacements as quickly and easily as possible.

Waterfall Agile – We’re supposedly “doing” agile but one or more of the following are true:

1. There’s a fixed deadline, with fixed features and fixed resources.
2. All Negotiation/Trade-off is done prior to project commencement with no review between sprints.
3. All sprints have been planned out in advance right up to the release date with no spare time.
4. There are no risks to manage (because there aren’t any apparently).
5. No-one is entertaining the idea of unknowns.
6. When a sprint doesn’t deliver as anticipated, outstanding work is simply crammed into the remaining sprints.

1. It’s quicker for me to write the code than explain the design to someone else.
2. Automated deployment will have to wait until we have more time.

Number one costs a software development team in a number of ways:

• The career development of other members of the team is slowed – if one never discusses design how does one expect to obtain good designers or architects?
• The team’s development capacity is reduced – essentially projects bottleneck around the uncommunicative heroic individual.
• The team’s effectiveness is reduced – project load cannot be divided efficiently because individuals have skills in narrow areas limiting the breadth of work they can perform.
• Team morale is damaged with other developers feeling left out, unfulfilled and unable to influence project decisions

Number two yields costs including:

• We save on some development time but the cost is re-surfacing in staff-hours required to perform the deployment.
• An increasing number of mistakes that extend deployment time or breaks releases.
• We save development time once and pay the price for that saved time with each and every deployment.
• The cost of each successive deployment increases because the system’s size is growing.
• As each deployment takes ever longer, the gap between releases is likely to increase.

Hyperchromatic trees are lazily balanced i.e. balanced in the background rather than as part of the operation that changes the structure of the tree and they have some challenging locking disciplines to implement as well. They are designed to perform better under concurrent access than their ancestors.

Looking at my notes I was running a single test performing concurrent operations continuously for periods of 5 days against an instance containing 1 million entries on JDK 1.4. I suspect this period of my developer life taught me more about deadlocks and race conditions than any other.

Technorati Tags: concurrency, development, software

We make this mistake all the time with languages, frameworks, applications and operating systems. When will we learn the lesson I wonder?

Technorati Tags: philosophy, development, software

Almost from the get go these days you’ll automate as much of this drudgery as possible using computers, essentially creating a one person IT department that supports your business processes. Obviously the bigger the business gets the more automation you will require, perhaps developing custom software to support the execution of your business processes (all that drudge you have to do but isn’t core). Your IT department will have to grow accordingly, hiring developers, project managers etc.

This standard tale of the relationship between business and technology plays out in many an enterprise every day. IT and in particular software development is a necessary evil, an undesirable cost to be carefully managed.

However for a certain class of enterprise, software is the product. Maybe it’s a shrink wrap product or a set of services provided to customers via the web. There will still be a need to have an IT department to support the business processes and it might well develop software of it’s own. There’s a fundamental difference between these two types of software because one of them is the business’ money generator. This software must be nurtured and tended to carefully, one can’t just add features, one must care about stability, performance, availability, scalability and so on. Money put into this effort is not a cost it’s an investment. Fundamentally, this software is not enterprise IT as most know it.

This key difference is oft-missed by management, vendors and analysts leading to confusion and unnecessary mistakes. I’ll cover one such mistake (related to SOA) in a follow-up post.

Technorati Tags: business, engineering, software

1. Agreeing what will be done
2. Agreeing how much it will cost
3. Agreeing a deadline by which the what will be done

Because these questions require that one knows everything down to the deepest detail and that all possible happenings are known (which means knowing when people will be ill and for how long, how much time they’ll need to take off for dealing with family troubles, problems with the plumbing etc) and the risks are mitigated such that they absolutely will not affect your project in unpredictable fashions.

That’s not to say that one can’t set deadlines but one has to expect to trade features away, adjust resourcing etc. Of course none of this is news and yet so many places claim to be agile whilst continuing to have the what, how much, when discussion.

Technorati Tags: development, engineering, process, software

Trouble is every decent software engineer knows that actually you want less code. It’s easier to maintain, easier to change, easier to debug, easier to build on. They’ll also tell you that the best way to build big codebases is out of lots of small, well isolated, loosely coupled bits with minimal knowledge of each other.

The less code philosophy requires doing some design, pausing for thought and not cranking code. It can provide massive benefit but it’s difficult to measure in any simplistic fashion and thus is seen as pure cost by many.

More code is the hare, less code is the tortoise – know which one I’d bet on.

Technorati Tags: development, software, systems

I believe the software we produce should be viewed as a stakeholder in it’s own right. It has it’s own needs for survival and ongoing growth and if these are always placed behind everyone else’s (i.e. the business) considerations, the results will be a long slow, painful death where the software becomes more and more brittle, less and less maintainable and staff productivity drifts down as staff turnover creeps up.

Consider a car – it has needs, new tyres, oil flushes, new exhausts, paint chip removal, new springs, new clutch etc. Fail to address those needs and your car will turn into a pile of rust before your eyes with all the attendant issues of depreciation, lost investment, breakdowns etc.

Why do we refuse to accept that software has a need for the motoring equivalent of oil changes and the like? Probably because software is an invisible abstract thing such that only those working with it see the damage being done, problem cars are easier to spot for a greater percentage of the population. This isn’t really an excuse however because software gives warning signs just as cars do. If there’s steam coming from under the bonnet (hood) you’d go to a mechanic, if your software keeps crashing or upgrades keep failing or development takes longer and longer it surely follows that it’s time to visit with your software engineer.

[ You may have noticed I like car analogies - here's another: You can't endlessly tune a car, it will only go so fast. Any further increase in speed can only come from starting with a new car that has better basic performance ingredients. The same is true for software, eventually you need a redesign to make further progress because the initial assumptions you made have all been invalidated. ]

Technorati Tags: development, software, systems

1. Google seems to have applied N > 1 to everything, not just storage – that’s pure distributed thinking, not the norm for the majority of software heads.
2. Eventual consistency might be kinda like concurrent programming for most people – i.e. many like to program sequentially and with the certainty that x has been completed immediately, in order and within a known timespan. Concurrency, eventual consistency and friends aren’t terribly amenable to this programming approach and it consequently melts many a brain.
3. We need for a lot more people to understand CAP.

[Note for Bill should he read this: it seems like your comments are broken, I'm seeing server errors when I hit post]