Why Agile will never be a project management framework
This is not a Waterfall-versus-Agile discussion:
This post is not about Waterfall-v-Agile. It is much wider than that. In its pure sense, Waterfall is a very specific and rarely used software development life-cycle. This post is about the realities of projects in business, regardless of how unpalatable those realities may be, and what the real world of projects is often about.
At its most basic, a genuine project is a unique problem to be solved, within the desired timeframe and within a specific budget. The fact that many human beings do not find meeting all three constraints together easy does not mean that we can simply ignore the ones we least like “This is Agile – we don’t do dates“. To be honest, that’s what many Agile ‘purists’ are trying to do.
Genuine projects do and will always exist. Agile is a product development approach (it is not a methodology) and there is no single “Agile” (1) (as in the noun). It is a collection of values and principles about software development. Most importantly, it is a huge mistake to call it or think of it as a project management framework. Also, Agilists frequently label every “non-Agile” approach as Waterfall. This is a gross over-simplification and misinterpretation of project management frameworks, few of which have very much in common with Waterfall.
Agile contains some great ideas, but sadly there are many who have hijacked Agile to promote their own beliefs about software development, especially their dislike of anything relating to the word ‘management’.
Good practices not unique to Agile
Many Agile purists talk as if the following only ever happens in Agile, or were invented by Agile. Neither is true as all the following have been employed on well-run (non-Agile) projects for decades (well before the Manifesto in 2001):
- Cross-functional teams with decision-makers embedded in the team.
- Face-to-face communication (on all key issues and questions).
- Use of simple to develop and update visuals.
- Daily communication around issues and current work.
- Use of iterations, mock-ups and early demonstrations.
- Early and frequent end-user review of deliverables.
All of the above help to minimise re-work and issues on projects and hence speed up the delivery process, while increasing confidence that the deliverables will meet users’ real needs.
When Agile works well
Firstly, there are some excellent results using Agile from people who interpret and implement Agile approaches skillfully and in a highly pragmatic way. They also recognise when they are in a project situation, and combine elements of Agile ways of working for the product development with whatever else is required to manage the whole project. Not in a command-and-control style, but by using effective ways of managing anything outside of the immediate horizon that Agile uses, and outside of the software development itself. And for those who say “anyone in the team will just pick this up and do this” – they are in urgent need of a reality check.
The Agile temptation
Agile vendors draw people in by the promise to deliver products quicker. There are few Executives in the world who are not going to want this. Often this is within the part of the business (Software and Information Systems) that has been notoriously problematic historically (relative to delivery) with a track record of routinely exceeding project budgets. Put these factors together, and Agile becomes a very easy sell, almost regardless of the price tag. However, there are some highly questionable tactics by some leading Agile vendors, as described below. Whole industries have been born around ‘new’ methods; as you cannot sell a set of values and principles.
Sadly far too often:
We deal with dozens of different organisations every year and Agile often comes up in discussions. In private, the most common thing we hear is “We moved to Agile; things are worse, not better”.
So let’s look at just some of the reasons why:
- Agile is easy to understand in principle but very hard to do in practice. Much harder than the Agile salesmen will admit. Most of the true benefits of using Agile are not commonly understood and the challenges involved often go unresolved. Culturally, they are some of the greatest challenges we can ever face in business. It will require a great deal of change to ways of working, current practices employed by developers and people in the business and even roles. It is often referred to as transformation but transformation efforts have a very high failure rate. That’s not something the Agile salesmen usually highlight and the scale of change required is often vastly underestimated, especially for larger projects.
- It should never be an ‘all-or-nothing’ decision; Agile should be used on selected projects only. This, however, brings a further challenge. Teams new to Agile will need a great deal of support.
- Far too many people confuse Scrum with Agile. When they say they are using Agile, they really mean Scrum. The most common interpretation of Scrum is totally anti the principles and values of Agile.
- Scrum is suited to simple software development where incremental sequential development and deployment are clearly appropriate and teams are independent but that is far from always possible. Where design also involves elements other than software or complexity is high, the core of Agile may be very challenging to master – beyond the appetite of many organisations. Many Agile methods ignore Design, almost as if it does not exist.
- Getting the real benefits of Agile takes a great deal more than introducing Sprints and Scrum. It takes understanding and commitment from the business and from everyone in the development teams. It will take time to develop an approach that works for you too, and this will have to be adapted when used on multiple projects. Projects by their very nature have unique characteristics and trying to develop a single ‘method’ to deliver them is never going to work. That is not a message many businesses understand or welcome.
- Despite the above, the world is suddenly now full of Agile transformation experts and coaches (many of whom have gained accreditation after as little as two days of training).
- Agile can make you become very short-sighted, making longer-term planning challenging.
- Technical debt is common and in the medium and long term can be very damaging and costly.
- In some instances, project managers have been dispensed with, leaving a big gap when it comes to managing crucial aspects of delivering projects, such as dealing with stakeholders and any kind of forward planning and much more. Also, assuming just anyone can succeed in this role is a very flawed assumption.
- Projects are often full of dependencies and dependencies are bad in Agile, especially Scrum. This often leads to poor decisions inside iterations and can result in a false interpretation of progress.
- You constantly hear things like: “this is Agile – we don’t do dates” from the mantra promoted by many purists. To be fair to them, the Agile manifesto talks about quality and releasing software frequently, but is very careful to avoid making any reference at all to “on-time” etc. Some organisations, like Google, don’t have to work to dates. Products are released when they are in a fit shape to do so. Google rarely share any announcements about dates publicly (unlike the old days of Microsoft). But, not every firm has that luxury. There are times when dates matter and in those circumstances, the schedule needs to be a constant and core consideration. The whole topic of schedule is deliberately avoided by far too many Agilists.
- The applicability of Agile is far less than the Agile salesmen claim.
- Some of the most valuable aspects of Agile can face push-back from developers who have little interest in anything outside of development using statements like “Stand-up meetings are an annoying interruption. We are developers. We write code.”
Also, the key test is not whether Agile is working for the development teams, it is whether Agile is working for those who fund those development teams, i.e. usually their customers (internal or external) and the users of the products being produced.
Too much of the good of Agile is damaged by dogma and questionable tactics:
There are highly professional organisations helping customers use whatever is right for the job at hand – and then there are those who will only ever suggest one approach regardless. They tend to be labelled as zealots. The latter often use some very unhelpful tactics and messages.
Here are just a few examples:
- Agile is a product development approach and Agile itself is not even a method – it is a set of values and principles. But, it is very hard to sell values and principles – hence the birth of many “Agile” methods.
- If you invent a whole new language to describe things you are already doing and pretend they are new, (2) you have the basis of selling something different and you create fear in people as if they don’t know the new language, they feel vulnerable. You have then created the perfect conditions for a whole new generation of certifications and the revenues these create.
- Anything “non-Agile” is frequently smeared as “command and control” or “traditional”. This could not be further from the truth and is either dishonest or born of a limited understanding of non-“Agile” approaches.
- In Agile, scope is the biggest variable. On many projects, regardless of product development method, this simply cannot be true.
- In Scrum, only the Product Owner is accountable for the projects’ success. In Scrum there is a refusal to accept any individual accountability apart from the Product Owner, stating: “transparency creates accountability”. Transparency is a great thing but teams without accountability are of very limited use. Also, accountability is not command-and-control and it can be generated in many positive ways.
- Failure to accept or even use the words “manager” or “leader”. This is dogma of the most damaging kind.
- Saying Agile can be applied to any project is like saying any person alive of any age can win an Olympic gold medal at the next games. There are some staggering examples out there. There are some projects where sequential incremental delivery is never going to be possible or applicable. The Manifesto is called “Manifesto for Agile Software Development” for a reason.
Conclusions:
- Agile can bring some very clear benefits to the product development element of some projects – but it (1) does not cover everything you have to do and cannot be used in all cases.
- It is not an either/or choice when deciding how best to deliver projects (i.e. whether to have project management or Agile) – we often need both, working together. Agile was never intended to be a project management framework (or method). It does not address large parts of managing projects, especially areas like integration and dealing with dependencies.
- The world of, or the word “project” are not going to go away, despite this being the desire of some people.
- The elements of project characteristics that many Agile “purists” dislike most of all “e.g. scope, schedule, timescales, budget” are realities of doing business and delivering work on behalf of stakeholders and customers. Doing business is not a hobby.
- Believing you are using Agile on some non-software projects is bordering on fantasy at times, or if you really are, there is a great chance it will end badly.
- It is rarely a straight choice between Agile and a more predictive form of delivery and if there is a very clear goal from the outset Agile may even bring more risk than reward.
- It is time for professional associations (such as PMI and APM) to show real leadership around the difference between project management and product development methods.
Notes/ References:
- Most references to “Agile” are in reality citing the use of specific methods, such as: Scrum, XP, Lean (is more of a principle than a method), Kanban and others. While these methods can have value, on their own they do not include a very large part of what is involved in managing and implementing complex projects. It is the need for those elements too that some people find very hard to accept.
- Just as one single example: “Planning Poker” – a method of estimating common in Agile – was derived from an estimating method called Delphi which was developed in 1944.
Share this:
23 Comments
Leave a Comment
Contact Us Now
EMAIL us today.
Or call ++44 (0)1865 784040
Nicely written, Kevin!!!
The only parts I really don’t agree with you are:
1) “genuine project is a unique problem to be solved”
A project is nothing more than a DELIVERY SYSTEM designed to “create, acquire, expand, update, maintain and eventually dispose of ORGANIZATIONAL ASSETS. (Defined to be Physical, Financial, Knowledge/Information, Human and Intangible.) Your definition is very limited.
2) “Agile is a product development approach (it is not even a method at all)”
I totally disagree with you on that point. “Agile” is nothing more than a new name for the “Scientific Method” http://mrbowlesscience.weebly.com/the-scientific-method.html
3) “The applicability of Agile is far less than the snake-oil salesmen claim.”
Not even close to being true. The taming of fire (1 million years ago) and the invention of the wheel (3500 BC) are both early examples of “Agile” or experimental models. Other more recent examples of “Agile” (Scientific Method) at work are:
Alexander Graham Bell’s invention of the phone;
Thomas Edison’s invention of the lightbulb;
Salk and his polio vaccine?
Flemming and Penicillin?
Need we go on???
But what makes your paper really SPECIAL in my eyes is because “It is time for professional associations (such as PMI and APM) to show real leadership around the difference between project management, and product development.”
Unfortunately, IMPO, the leadership of both PMI and APM has been lacking for many years as their primary focus has been from improving the practice of project management in favor of building up their piggy banks by selling largely useless exam based certifications. Let’s hope this changes.
BR,
Dr. PDG, Jakarta
Agile is presented as an alternative to traditional project management methodologies (PMM). However, my impression is that Agile mischaracterizes PMM, which undermines Agile’s credibility.
Agile principles are presented as though they are differentiators, but they’re also part of PMM. Principles common to both include: deliver highest value items first, identify issues early and often, don’t burn people out, provide technical excellence, assess processes and outcomes, continuous improvement, recognize that things change frequently in project environments, focus on business priority and customer value, extensive customer involvement, team empowerment, and use of cross-functional teams.
Agile presents values as though they are competing, but they’re not. We don’t have to choose from among them. Effective projects value individuals and provide processes and tools. They value working software and comprehensive documentation. Effective project teams collaborate with customers and are aware of contractual requirements. Project teams value responding to change and following a plan.
I also disagree with Agile assertions that in PMM:
• You have to completely finish one phase (requirements, design, development, testing) before you move on to the next.
• Customer feedback occurs late, and changes are addressed only after testing so that changes are difficult, and the customer sees the product only once it’s fully completed.
• Teams are top-down, heavy direction groups where people just “take tasks from a PM”.
• The methodology is prescriptive (narrow, rigid, imposes rules. (It’s customizable and scalable)
• There’s little customer involvement after development starts until after the product is complete.
• The focus is on comparing actual to baseline.
• Requirements are locked down with no communication until delivery, which creates risk that a team might deliver the wrong thing. (Effective projects use requirements reviews, functional and technical design reviews, release management, frequent customer engagement, customer developed test cases, end-user testing, and requirements traceability matrices to provide product quality assurance. It’s never a surprise at the end).
Hi Wes – many thanks for your comments – I agree with everything you say under “I also disagree with Agile assertions that in PMM ……”. Well managed projects (Agile or non-Agile) never do these things and to claim that this is how all non-Agile projects happen, is just totally untrue.
Wes, I don’t agree that Agile is a PART of PMM. They are, IMPO, two very distinct and DIFFERENT approaches that are EQUAL (one is not a subset of the other) and although they may use some common tools & techniques, the METHODOLOGIES are very different.
Agile is, as I hoped I explained above, is nothing more than a variation on the “scientific method” which dates back to the 12th Century (Magnus, Aquinas and the two Bacons, Roger and Sir Francis et al) Agile works VERY well when the objective is not known in advance or is subject to significant changes, while the pretty much linear project management processes (Initiate, Plan, Execute, Control and Close) are perfectly adapted to a situation where the objective or end result is either known in advance or knowable as the processe evolves and is not subject to major changes.
Where I fully agree with Kevin is that the two processes should NOT be conflated with one another and that the responsibility for making that clarification lies primarily with PMI and APM in particular.
BR,
Dr. PDG, Jakarta, Indonesia
I agree with Dr. Paul D. clarification above where he made the distinction on applicability of Agile and Project Management Processes.
I have used and think that Agile is a development/progression methodology which can very well be used under each phase of project management lifecycle. Whoever discard this fact and promote agile at the expense of PM Processes, anyways use other inefficient approaches of gathering resources, collecting and elaborating requirements & design and other important PM Processes. They don’t account these important aspects of a project. I have seen many projects fail, starting as agile and later after a year or 2 were closed down because the product vision didn’t satisfy the business requirement as they were either dragged on because of ever increasing scope or tech-debt.
It is better to articulate before hand the vision of the product, its timeline and budget, if it is to be used as a successful ROI.
Oddly enough I just wrote a LinkedIn article about “development” vs. “delivery” and the use of the word “methodology”. Much of what you are saying lines up with my recent article.
What we usually hear as a “development methodology” is actually a “delivery approach or framework” with the focus on providing “value” when something is released into whatever production/operations looks like. The mechanics of building the “whatever” doesn’t change just because you label it “agile” as you noted.
Very interesting food for thought that continues the push I and others make to debunk the mysteries being created by the ones I label (in my series of articles) the Counterfeit Ring.
Mark got the link to your article, please?
BR,
Dr. PDG, Jakarta
Sorry for the tardy reply, Paul. Here it is:
https://www.linkedin.com/pulse/busting-counterfeit-ring-words-important-mark-moore-m-ed-pmp/
Mark
Hooray! Its nice to see people standing up to the “Smoke and Mirror” travelling salesmen who constantly bring out this garbage to sell to insecure business executives who, being totally ignorant of what they are buying, see only the golden bullet of the Month – They think it shows they’re ‘doing something’ to their bosses and shareholders – pity the something is just burning money needlessly. Like all things you need to look at AGILE in context, yes it has its little list of methodologies / practices (whatever words you like to label it with) that are applicable in a small number of cases – but the real world doesn’t work this way, change happens all the time and at an infinite number of different levels – so if you are only making small application changes then you are doomed to failure as major change engulfs you and plows you under. Anyone who thinks AGILE, as sold by the snake-oil salesmen, is going to solve ALL their IT problems is a fool! and thats what they will look like when it all fails.
Agile is not a framework; it’s a mindset distilled from a collection of frameworks, so the main topic’s debate is moot on that front.
As for Agile proponents ‘simply ignore the ones (scope, time, budget) we like least’, you fail to back your accusation with any evidence whatsoever.
You then go on to claim ‘Many Agile purists talk about it as if the following only ever happens in Agile, or were invented by Agile’… now I have yet to encounter any Agile professional who claims that ‘Agile’ invented anything. Agile simply consolidated the best (software and manufacturing) practices from the previous century.
Richard – thanks for your contribution:
– Agile itself is not a mindset – it is a set of values and principles – it is possible for mindsets to follow these but not common.
– in terms of evidence, I would invite you to do a number of things – follow the links in this post as a start including the link to the discussion about scope and schedule etc. Clear evidence I would say.
“Agile is a mindset” is an utterly meaningless marketing phrase used by snake oil salespeople.
great article – “It is time for professional associations (such as PMI and APM) to show real leadership around the difference between project management and product development methods”. – that says it all
As someone who is SCRUM certified and who has worked on (and witnessed) numerous so-called “Agile” based projects using the SCRUM Framework across a number of different companies in different industry sectors, I would not recommend the use of the SCRUM Framework for Project Management, Product Development, Software Development/Engineering or Engineering in general.
SCRUM is simply not fit-for-purpose for anything. It is a shallow framework and is not a rigorous or robust engineering standard or methodology or best practice. It does not give any engineering guidance to design, engineer or deliver a project or product or system or software. It cannot be used to give engineers engineering guidance in how to design and engineer from concept now products and/or systems. For this, we fortunately have to rely on INCOSE’s Systems Engineering Handbook where it is systemic practice to follow the section on Lean Systems Engineering which is the most Lean and Agile approach out there, in fact, it’s Agile on steroids.
SCRUM is for the uneducated engineer, i.e., for people who despite their level of education just don’t know how to design and engineer new products/systems/software (and would be great for high-school level software development) where they don’t care about taking a systems approach to work out what the right solution should be for a given problem, but would rather just hack something together in bi-weekly sprints and just hope for the best (and if you’re not doing this, then, you’re not doing SCRUM). Why do you think there are no roles in SCRUM for Requirements Engineers, Systems Engineers, Systems Modellers, Systems/Software Architects and so on?
In conclusion, SCRUM is great if you want to do cowboy-engineering and produce something or anything in a quick ‘n’ dirty way just to please someone, it is not a tool for any serious hard-core Engineer, Project Manager, Product Manager or for someone remotely intelligent. It will always be used by people who struggle to solve complex problems, have a poor ability to reason and articulate system complexity in abstract UML/SysML design and are looking for a quick ‘n’ dirty way to produce working software. Any idiot can do this, the question is whether it is right or wrong. A Product Owner on his/her own cannot possibly do this as they only cover their perspective/agenda and view of the world, but there are other perspectives and it is impossible for one single person to give the overall vision and requirements for highly complex-systems. It is why we use and rely on Subject Matter Experts and other key stakeholders. Common sense alone should tell you this. Do we really need to debate this? I would not recommend or use SCRUM for anything which I care about, unless, I was forced to by my customer or the environment I’m working in which would be unfortunate. SCRUM, like previous ideas will just fall to the way-side and will not survive the test of time. I’m also glad that it is not being taught on Software Engineering degrees.
Please don’t hold back Mr. Tetlay. Tell us how you really feel about anyone using something you don’t agree with, not just SCRUM. I’m sure all of us who are far from intelligent would benefit greatly from your insight.
As you’ve learnt the hard way; I never do. I just have. Yes, of course and naturally. It’s one of the reasons why I work as a consultant.
Totally ignorant of what agility really is. But, not surprising.
Thank you for your reply – I have been applying approaches that are common in Agile since well before 2001. My article is referring to the interpretation and promotion of Agile by many people today and the way vested interest has hijacked much of Agile, not the Manifesto or the better interpretations of Agile. There are many references within the article to support my comments, demonstrating I am far from alone.
On a more general note, the great news is that we live in a world where I have the right to express my views and I usually do so without calling others “ignorant”. Lastly, I have shared a post of substance, that is backed up by real-world experience and references.
The truth hurts sometimes; I know. You shouldn’t hate what you don’t understand and you shouldn’t refer to it as ignorant as that’s arrogant, and especially, when you don’t have the intelligence to articulate a counter argument, but, I’m not surprised with your level of intelligence which is typical behaviour from people who champion agility.
Define Agile project. Please cite where the Agile Manifesto defines an Agile project. I also suggest re-reading the Scrum Guide.
Clyde – thank you for your comment. Don’t think I have used the term Agile project in this post. Also, I have read the Scrum guide many times and and have written on Scrum here. This second post has a number of references to support the points being made and I encourage anyone to read them.
Its always interesting to read the comments after someone dares to challenge Agile, in any context.
The article itself is fair and makes a number of valid points, particularly around the misconception that PM methodologies dont contain similar principles to Agile and the saturation of Agile practioners.
This second point is a result of a practice IT does extremely well…monetising knowledge. Often pure intentions for improvement are lost in the search for profits. This is obviously not just an IT trait, but I think we are probably the best at it. There is certainly nothing wrong with a business making money, but the race for market share of certified practitioners inevitably releases a few less than capable parties into the wild.
The main problem with IT in general, is the constant combativeness between opposing factions (as illustrated in the comments). There is often no allowance for discussion as both parties ‘know’ unequivocally that they are right.
This again is not limited to PM/Agile, it exists in every area of IT from infrastructure products to service mangement techniques.
The reason the majority of IT projects are deemed to have failed (in some respect) is simply due to a lack of collaboration between delivery personnel.
Agile to its credit tries to highlight the importance of collaboration in achieving delivery goals.
Overall as in anything, it is the abilities of the people performing the activity that will determine success, irrespective of the approach being utilised.
thanks for your comments Alan. I agree particularly with your comments on lack of collaboration etc. All I would add is that effective collaboration is more about behaviours of individuals and groups rather that the adoption of any specific method in a zealot type of manner.