Balancing Agility and Discipline1 Balancing Agility and Discipline: A Guide for the Perplexed; Boehm & Turner; Addison–Wesley; 2003. talks a lot about the contrast between agile development and plan-driven development. We will briefly look at the contrast at the end of this article, but first an insight that I think is much more fundamental.
Two scales of plan-driven development
There are really two different things called plan-driven development, and almost everyone – including the authors – conflate the two.
Product development is about collecting usable knowledge. Both types of plan-driven development tries to make this hard-earned knowledge explicit using written documentation. This is in contrast to agile development which aims to build tacit knowledge in the brains of the people working on the project.
Looking at what these types of plan-driven development has in common misses the fact that they are very different. The book never states this explicitly, but their difference emerges between the lines.
The first thing is captured by the authors when they say,
[Plan-driven development] fitted well with the branch of academic computer science and software engineering that views software development as a process of formal mathematical specification and verification.
This is a statement about the implementation of specific modules, or specific algorithms. Plan-driven development in this context means writing a detailed specification of exactly what the module should do, converting that to detailed design diagrams saying exactly how the module should do it, deriving acceptance tests from the design, and then passing the design off to programmers for implementation. In other words, a very small, clearly bounded waterfall.
Then the authors switch to talking about something else, seemingly without even noticing it themselves!
Plan-driven methods rely heavily on documented process plans (schedules, milestones, procedures) to keep everyone coordinated. Considerable effort is spent on maintaining historical data so that planning projections can be more accurate. Plans make explicit the expectations and relationships between the project and other independently evolving systems.
Now we are no longer talking about individual modules. Now we are discussing coordinating the effort of hundreds or even thousands of humans toward some shared goal! In this context, plan-driven development is about up-front sequencing the development of modules, personnel planning, predicting budgets and time, issuing detailed status reports, and so on. It’s about knowing where we are, where we’re going, and how long it will take to get there.
We don’t have to choose just one
When is it beneficial to do a plan-driven implementation, and when should we remain agile in implementation? When is it beneficial to set the project direction by plan, and when is it best to remain agile toward the future? Those questions actually beside the point for this article, but we can look at some reasoning in the next few sections. First, let’s re-emphasise the core insight.
- We can do plan-driven development (or stay agile) for each individual module implementation.
- We can also be plan-driven when we set the direction of the entire project (or stay agile).
Those two are not related. It’s fine to have a plan-driven project direction and only agile implementations. It’s also fine to have an agile project direction with every implementation being plan-driven.
This releases us from the agile/planning dichotomy for projects: we can actually use both, and most projects probably should.2 Even the most innovative software has a few routine parts, and even the most boring software has a few novel parts – otherwise we would just reuse existing software.
Plan-driven or agile implementation?
What does Boehm have to say about when one should switch from agile to plan-driven implementation?3 Technically, Boem says plan-driven should be the default and agile the odd one out. I’m not ready for that yet. Here are some signs that we can benefit from plan-driven implementations:
- It is well specified what the component should do.
- Requirements are either stable or change in predictable ways.
- The end product has the potential to save (or take) many lives.
- The product only integrates with familiar technologies.
- Non-functional requirements4 Performance, security, reliability, etc. are clearly articulated and relatively easy to satisfy.
- We have some people on the team who can write complete requirements specifications and create software design diagrams and analyse them for problems.
If many of these criteria are fulfilled, plan-driven development gives us primarily two things:
- We can pawn off the actual programming work to cheaper, less skilled programmers.
- We can get some assurance that the end product is exactly what was intended.
If few of the above criteria are fulfilled, we should probably stay agile in the implementation of the component. Note, however, that with agile methods, we need a large proportion of skilled programmers on the team, and we don’t get any assurance that the end product is exactly what was intended.
You may be confused about the grey area. We have talked about when many of the criteria for plan-driven implementation are fulfilled (go plan-driven) and when few of them are (go agile). What if only some of them are? You may be in deep shit. If we don’t have people who are good at planning and our requirements change unpredictably, yet our product can potentially kill many people, that is a good time to take a step back and think about whether the product needs to be built at all.
Plan-driven or agile direction?
That’s implementation. What about running the entire project in a plan-driven way, or agile? The main benefit we get from plan-driving projects is that we can control – or at least observe – the progress of the project from start to finish with some precision. A plan-driven project is one in which the success metric truly is “percent of plan delivered.” This precision lets us e.g. move in lockstep with another organisation, or align investment with recurring payments against specific deliveries.
Another benefit of coordinating through written documentation, according to the authors, is that it allows coordinating larger numbers of people. They claim you can have hundreds of people contributing to the same goal simultaneously, by distributing the right documentation at the right time. The specific people on the project also matter less – anyone who can read and interpret the documentation can fill the seat of someone who leaves.
That sounds great! When can we do it?
- We need to be able to know the end goal of the project in advance.5 This is surprisingly rare, because it means we cannot be victim to market whim, which most software product development organisations are. But it may be the case e.g. if we are re-creating something that already exists but changing some underlying technology, or building a well-specified safety-critical system, or repeatedly making very similar products (e.g. content management systems) for different customers.
- We must be willing to pay extra up front for the benefits, including increased time-to-market. Plan-driven development does cost more and take longer up front, but if the benefits earn us this money back and then some, it’s an easy choice.
- We need to have a sufficient number of people who are skilled at documenting, estimating, tracking, reporting, and so on. You may or may not think of these as traits of a typical developer.
These are big asks, but if you find a project where it is true, go plan-driven!
Don’t forget about the people
All of this has been about technical considerations. One also has to consider what the programmers themselves prefer. Someone who thrives on policies and processes and clearly delineated responsibilities will flop around like Magikarp in an agile effort. Someone whose creativity is unlocked by uncertainty and open doors will be hemmed in by a plan-driven effort.
This is perhaps the overriding concern. As good a technical fit as a project is for plan-driven development, if the developers don’t know what to make of it, it won’t be successful. Either they need to be trained, or the project has to be scaled down to fit an agile, interpersonal approach.