Why Agile will never be a project management framework
This is not a Waterfall-versus-Agile discussion:
This post is not about Waterfall-v-Agile. It is much wider than that. In its pure sense, Waterfall is a very specific and rarely used software development life-cycle. This post is about the realities of projects in business, regardless of how unpalatable those realities may be, and what the real world of projects is often about.
At its most basic, a genuine project is a unique problem to be solved, within the desired timeframe and within a specific budget. The fact that many human beings do not find meeting all three constraints together easy does not mean that we can simply ignore the ones we least like “This is Agile – we don’t do dates“. To be honest, that’s what many Agile ‘purists’ are trying to do.
Genuine projects do and will always exist. Agile is a product development approach (it is not a methodology) and there is no single “Agile” (1) (as in the noun). It is a collection of values and principles about software development. Most importantly, it is a huge mistake to call it or think of it as a project management framework. Also, Agilists frequently label every “non-Agile” approach as Waterfall. This is a gross over-simplification and misinterpretation of project management frameworks, few of which have very much in common with Waterfall.
Agile contains some great ideas, but sadly there are many who have hijacked Agile to promote their own beliefs about software development, especially their dislike of anything relating to the word ‘management’.
Good practices not unique to Agile
Many Agile purists talk as if the following only ever happens in Agile, or were invented by Agile. Neither is true as all the following have been employed on well-run (non-Agile) projects for decades (well before the Manifesto in 2001):
- Cross-functional teams with decision-makers embedded in the team.
- Face-to-face communication (on all key issues and questions).
- Use of simple to develop and update visuals.
- Daily communication around issues and current work.
- Use of iterations, mock-ups and early demonstrations.
- Early and frequent end-user review of deliverables.
All of the above help to minimise re-work and issues on projects and hence speed up the delivery process, while increasing confidence that the deliverables will meet users’ real needs.
When Agile works well
Firstly, there are some excellent results using Agile from people who interpret and implement Agile approaches skillfully and in a highly pragmatic way. They also recognise when they are in a project situation, and combine elements of Agile ways of working for the product development with whatever else is required to manage the whole project. Not in a command-and-control style, but by using effective ways of managing anything outside of the immediate horizon that Agile uses, and outside of the software development itself. And for those who say “anyone in the team will just pick this up and do this” – they are in urgent need of a reality check.
The Agile temptation
Agile vendors draw people in by the promise to deliver products quicker. There are few Executives in the world who are not going to want this. Often this is within the part of the business (Software and Information Systems) that has been notoriously problematic historically (relative to delivery) with a track record of routinely exceeding project budgets. Put these factors together, and Agile becomes a very easy sell, almost regardless of the price tag. However, there are some highly questionable tactics by some leading Agile vendors, as described below. Whole industries have been born around ‘new’ methods; as you cannot sell a set of values and principles.
Sadly far too often:
We deal with dozens of different organisations every year and Agile often comes up in discussions. In private, the most common thing we hear is “We moved to Agile; things are worse, not better”.
So let’s look at just some of the reasons why:
- Agile is easy to understand in principle but very hard to do in practice. Much harder than the Agile salesmen will admit. Most of the true benefits of using Agile are not commonly understood and the challenges involved often go unresolved. Culturally, they are some of the greatest challenges we can ever face in business. It will require a great deal of change to ways of working, current practices employed by developers and people in the business and even roles. It is often referred to as transformation but transformation efforts have a very high failure rate. That’s not something the Agile salesmen usually highlight and the scale of change required is often vastly underestimated, especially for larger projects.
- It should never be an ‘all-or-nothing’ decision; Agile should be used on selected projects only. This, however, brings a further challenge. Teams new to Agile will need a great deal of support.
- Far too many people confuse Scrum with Agile. When they say they are using Agile, they really mean Scrum. The most common interpretation of Scrum is totally anti the principles and values of Agile.
- Scrum is suited to simple software development where incremental sequential development and deployment are clearly appropriate and teams are independent but that is far from always possible. Where design also involves elements other than software or complexity is high, the core of Agile may be very challenging to master – beyond the appetite of many organisations. Many Agile methods ignore Design, almost as if it does not exist.
- Getting the real benefits of Agile takes a great deal more than introducing Sprints and Scrum. It takes understanding and commitment from the business and from everyone in the development teams. It will take time to develop an approach that works for you too, and this will have to be adapted when used on multiple projects. Projects by their very nature have unique characteristics and trying to develop a single ‘method’ to deliver them is never going to work. That is not a message many businesses understand or welcome.
- Despite the above, the world is suddenly now full of Agile transformation experts and coaches (many of whom have gained accreditation after as little as two days of training).
- Agile can make you become very short-sighted, making longer-term planning challenging.
- Technical debt is common and in the medium and long term can be very damaging and costly.
- In some instances, project managers have been dispensed with, leaving a big gap when it comes to managing crucial aspects of delivering projects, such as dealing with stakeholders and any kind of forward planning and much more. Also, assuming just anyone can succeed in this role is a very flawed assumption.
- Projects are often full of dependencies and dependencies are bad in Agile, especially Scrum. This often leads to poor decisions inside iterations and can result in a false interpretation of progress.
- You constantly hear things like: “this is Agile – we don’t do dates” from the mantra promoted by many purists. To be fair to them, the Agile manifesto talks about quality and releasing software frequently, but is very careful to avoid making any reference at all to “on-time” etc. Some organisations, like Google, don’t have to work to dates. Products are released when they are in a fit shape to do so. Google rarely share any announcements about dates publicly (unlike the old days of Microsoft). But, not every firm has that luxury. There are times when dates matter and in those circumstances, the schedule needs to be a constant and core consideration. The whole topic of schedule is deliberately avoided by far too many Agilists.
- The applicability of Agile is far less than the Agile salesmen claim.
- Some of the most valuable aspects of Agile can face push-back from developers who have little interest in anything outside of development using statements like “Stand-up meetings are an annoying interruption. We are developers. We write code.”
Also, the key test is not whether Agile is working for the development teams, it is whether Agile is working for those who fund those development teams, i.e. usually their customers (internal or external) and the users of the products being produced.
Too much of the good of Agile is damaged by dogma and questionable tactics:
There are highly professional organisations helping customers use whatever is right for the job at hand – and then there are those who will only ever suggest one approach regardless. They tend to be labelled as zealots. The latter often use some very unhelpful tactics and messages.
Here are just a few examples:
- Agile is a product development approach and Agile itself is not even a method – it is a set of values and principles. But, it is very hard to sell values and principles – hence the birth of many “Agile” methods.
- If you invent a whole new language to describe things you are already doing and pretend they are new, (2) you have the basis of selling something different and you create fear in people as if they don’t know the new language, they feel vulnerable. You have then created the perfect conditions for a whole new generation of certifications and the revenues these create.
- Anything “non-Agile” is frequently smeared as “command and control” or “traditional”. This could not be further from the truth and is either dishonest or born of a limited understanding of non-“Agile” approaches.
- In Agile, scope is the biggest variable. On many projects, regardless of product development method, this simply cannot be true.
- In Scrum, only the Product Owner is accountable for the projects’ success. In Scrum there is a refusal to accept any individual accountability apart from the Product Owner, stating: “transparency creates accountability”. Transparency is a great thing but teams without accountability are of very limited use. Also, accountability is not command-and-control and it can be generated in many positive ways.
- Failure to accept or even use the words “manager” or “leader”. This is dogma of the most damaging kind.
- Saying Agile can be applied to any project is like saying any person alive of any age can win an Olympic gold medal at the next games. There are some staggering examples out there. There are some projects where sequential incremental delivery is never going to be possible or applicable. The Manifesto is called “Manifesto for Agile Software Development” for a reason.
- Agile can bring some very clear benefits to the product development element of some projects – but it (1) does not cover everything you have to do and cannot be used in all cases.
- It is not an either/or choice when deciding how best to deliver projects (i.e. whether to have project management or Agile) – we often need both, working together. Agile was never intended to be a project management framework (or method). It does not address large parts of managing projects, especially areas like integration and dealing with dependencies.
- The world of, or the word “project” are not going to go away, despite this being the desire of some people.
- The elements of project characteristics that many Agile “purists” dislike most of all “e.g. scope, schedule, timescales, budget” are realities of doing business and delivering work on behalf of stakeholders and customers. Doing business is not a hobby.
- Believing you are using Agile on some non-software projects is bordering on fantasy at times, or if you really are, there is a great chance it will end badly.
- It is rarely a straight choice between Agile and a more predictive form of delivery and if there is a very clear goal from the outset Agile may even bring more risk than reward.
- It is time for professional associations (such as PMI and APM) to show real leadership around the difference between project management and product development methods.
- Most references to “Agile” are in reality citing the use of specific methods, such as: Scrum, XP, Lean (is more of a principle than a method), Kanban and others. While these methods can have value, on their own they do not include a very large part of what is involved in managing and implementing complex projects. It is the need for those elements too that some people find very hard to accept.
- Just as one single example: “Planning Poker” – a method of estimating common in Agile – was derived from an estimating method called Delphi which was developed in 1944.