Why Scrum alone does not work

and, how to make it work much better:

The purpose of this post is to describe the limitations of (just using) Scrum and then later we describe what would work much better, while still using incremental product development. In particular, in the second part we focus on how to move well beyond the mantra of delivering “value” to something much more specific, beneficial and measurable, collaboratively.

Agile is a collection of values and principles relating to software development.  A number of “Agile” methods have emerged (since 2001), the most common of which is Scrum.  However, Scrum has become controversial and technical forums are awash with feedback from developers who do not support the method, to put it mildly. But that is only part of the issue.

Our view is that some of Scrum reflects Agile principles but much of it is anti-Agile. Furthermore, we believe that Scrum is not enough for the majority of projects and especially when you are engaged in Enterprise-wide system development and deployment, or large-scale or complex system development.

The reason we say Scrum is anti-Agile is that the Scrum Guide is prescriptive and includes many rules, that it says must not be broken. Some of those rules lead directly to the issues that follow.

  1. Scrum focuses on product development via the product backlog. On vast numbers of projects, there are many other things (1) that have to be done that are outside of the product backlog and Scrum refuses to recognise these or assumes the Scrum team will do these things (while they are being judged by their velocity in relation to the achievement of items in the product backlog). However, Scrum defines a team as only being the “professionals” required to “produce a single product increment, while literally forbidding any other role to exist other than the Product Owner, outside of the core Scrum team (2). This is simply not real world. Scrum does not cover all you typically have to do (in projects) or recognise that you will need many other roles to deliver a successful project.
  2. Scrum is totally introspective. Not once, in the whole guide does it mention the needs of others outside the Scrum team. It totally avoids any relationship between the work of those outside the Scrum team, and Scrum teams.
  3. Scrum includes no Software Engineering practices and can easily result in technical debt, considerable code refactoring or even development being scrapped, once requirements are fully understood in later Sprints, making current software or designs no longer usable. Scrum almost ignores design and encourages emergent Architecture and Design. This adds further risk around technical debt, flexibility of Design around future requirements, and maintainability of developed code.
  4. Scrum encourages that deployable code is developed per sprint, but makes no mention of the relationship between deployment and updates to business processes and functions that are likely to be required/impacted.
  5. Scrum is inflexible and prescriptive (3) – it states there are only 3 roles allowed even though none of these roles are likely to pick-up and manage the items mentioned above in #1. It is full of “Rules of the game” which is always going to raise the likelihood of issues (and gaps) that really should not be happening.
  6. Scrum states it proves it is a vehicle for dealing with complexity. It says that it is ideal for dealing with complexity but offers zero comments on how to. In reality, following Scrum’s fixed model (“religiously”) will result in complexity issues, not resolve them.
  7. According to the guide, Scrum teams: are “highly flexible and adaptive”. They also need to be stable but it fails to mention this. Both of these requirements are very hard to achieve in the real world. That is the real reason why Scrum authors themselves admit (4) that large numbers of Scrum teams do not produce shippable product at the end of each Sprint. The other reason for this is that large numbers of Scrum teams fail to test inside Sprints, and those that do, often develop an ever growing “bug fix” backlog that is in addition to the product backlog. I recently (April 2018) heard of one (from a single project in a major corporation) that had over 3000 items in it. Yes, that is three thousand, yet their “velocity” was still based on stories supposedly “done”. No doubt claims of amazing new productivity are still being made, by some.  It is also true that in many Agile teams test plans are a thing of the past. Where this is true, it has a clear relationship with this whole issue.
  8. The guide makes some astonishingly sweeping statements about all Scrum teams: “Everyone focuses on the work of the Sprint and the goals of the Scrum Team”. While this statement sounds great, if we are being honest, if we are all being totally honest we know it is optimistic. As an underlying key principle and assumption, it leads Scrum to make some very bad decisions, especially around areas such as leadership. Scrum lacks leadership.
  9. No one would argue against having a product owner. However, outside of software houses where the core business is developing software, the expectations around the PO can be wildly optimistic (5). There are some key responsibilities on projects that Scrum places on them that they are rarely going to do in practice (6).
  10. Scrum can make you very short-sighted, focusing no further than the end of the next Sprint. It even uses an analogy of saying the next Sprint is the project (7). This is a very bad habit and risks creating an attitude among teams that anything beyond this horizon does not matter (yet). Projects often encounter long lead times, and should this be the case, this is just one reason why this “habit” is very damaging. Understanding requirements around sequential iterations can mean earlier decisions have to be re-visited. In some cases, that may be very difficult, costly or even not feasible.
  11. Scrum refuses to recognise roles such as “lead developer”. This de-motivates people and affects quality.
  12. Scrum quotes words like “openness, respect and professionals” while defining a framework that results in more micro-management than most people care to be subject to. Especially when you add the use (and misuse) of velocity. Add this to the frequency of sprints and many highly professional developers are going to see this as a very short-term environment for them. Burnout is going to happen very quickly.
  13. Scrum does not recognise dependencies (outside the Scrum team) and in any kind of development, it’s almost impossible to eliminate them. To ignore them is disastrous. Projects do not just consist of the software alone.
  14. Scrum Masters don’t lead dev teams but they do lead the Scrum process in a team. Anyone can get a Scrum certification after two days of training, regardless of what they did previously and call themselves a Scrum ‘coach’. Or even worse: an Agile transformation coach.
  15. Scrum and Agile are often described as a “religion” (8). A religion is something a group of people believes in, but no one can prove its basis. When any kind of belief starts to alienate people who are not from within, bad things usually follow.

What would work much better?

  1. Scrum (and many in Agile) throws the word “value” about (e.g. customer value) constantly. I have no idea what this really means, and I suspect I am not alone (but it sounds great, right?). Projects don’t start with a Product Backlog. They should always start with far more strategic elements, such as: the specific problem you are trying to solve; the project goal; and the outcomes that the project is intended to deliver.  These things should be articulated clearly around stakeholders like customers and should be measurable in as many cases as possible.  The whole thinking and assumptions must be validated (in a very “light touch” way) with the key stakeholders and especially those funding the project. This might be harsh, but this piece really does matter and goes much further than throwing around the word “value”. Also, all of this must happen before there are any commitments to the “solution” or assumptions could become very costly.
  2. Once you have decided you have a problem really worth fixing, you can explore ideas and options. We can weigh these up and make a decision on which way to go, involving the right people. This will be people from the business and technology leads and delivery managers. This is to ensure we have optimum ideas and we start to build a common understanding of what we are trying to achieve, not just what we are trying to build. We must always be focused on the outcomes we want to achieve, not just the products/solutions we want to build.
  3. Once we get to the point of settling on a solution, which may involve product development, we can start to explore what the product needs to look like and do. But, all efforts and decisions must be focused on maximising the outcomes we want to achieve. All those outcomes will be business outcomes, as that’s how we fund projects and more.
  4. As soon as possible, we should then involve people who have experience of design, architecture and/or systems engineering, depending on the exact nature of the solution. We should also involve a project manager who preferably has experience of this type of project. As early as possible, we should start to evolve an understanding of the full scope of work of the project, not just the product development. We should look at aspects such as design, integration, testing, data migration, business change and deployment at the whole project level to ensure there is coherence and validity to all aspects, especially integration and testing (two aspects where Scrum is weak). We should test all major assumptions as soon as we can. Those assumptions can relate to the solution or the outcomes the project is intended to achieve.
  5. We should develop a team, which includes all the key people (parties) responsible for all the major elements of the project, not just the product development. The product development element must not be managed in isolation; it must be managed as an integral part of the overall project. All people in the whole project team must be fully aware of the broader business needs of the project and the outcomes it is intended to achieve, and they should take these into account in all their decisions.
  6. Then, we should decide which development approach we can and should take. This may be Agile assuming Agile is a good fit for the project. If it is not, it will bring more risk than reward. The goal is to deliver a successful project in its entirety; not to adopt a single development methodology religiously. We must be pragmatic in terms of the approach and admit if the circumstances for any single approach will not work or are not appropriate. If Agile is a good fit, we should implement it flexibly and with skill but never in a prescriptive manner.
  7. Once we have made that decision, we can evolve a plan for the whole project. We may have a release plan by then which could form a key element of that plan. In developing this, we should involve all the functions involved in the project about their own element and what their needs are to be successful and what they need from others. We should make the overall plan highly visible, and address risks associated with any part of the project, to develop increasing confidence in and commitment towards the plan. We should make all responsibilities within the plan as crystal clear as possible and should it be necessary to revisit this question at any stage of the project.
  8. Regardless of whether we are using Agile or not, we should employ a cross-functional team, develop an environment for the project where issues are raised and highlighted as quickly as possible, have visibility and transparency around progress, and invite feedback from stakeholders as early as possible and as frequently as we feasibly can. We should involve as many people as we reasonably can in decision making too while ensuring there is clear leadership around the whole goals of the project, and most importantly the outcomes it is intended to achieve.
  9. We should employ product based planning at the heart of the project, again, regardless of whether using Agile or not. The configuration of the system should be highly evident (and central) using this approach, which will enable effective planning around each configuration item. The same can be said for integration, test and deployment planning. It will also provide the perfect vehicle to identify risk and develop a systematic risk management plan – possibly the single most valuable thing any project can have.
  10. Assuming we are using an Agile approach, all development planning (including release and iteration planning) must take place within the context of the overall project plan. There will be times when parties outside the development team will need to have input to iteration plans.  This is to respect the needs of the customer and the overall project. Having “rules” that prevent this do not reflect real-world needs of customers, stakeholders and delivering projects successfully (i.e. the business outcomes that underpin why the project is being done in the first place). If this can all be achieved realistically via the Product Owner, fine; if not other arrangments must be respected. In real Agile projects, the highest priority is delivering maximum value to customers, not “religious” adherence to fixed “rules”.
  11. The goal is not to execute any single development methodology; it is to deliver a project that achieves the outcomes the customer is looking for; and yes, dates matter and dependencies (outside of the product development) cannot be ignored in that process nor can we pretend they do not exist. Lastly, budgets matter too. Just like they would if it was our own money.

Lastly, why no diagrams or pictures in this post?

Because, as soon as you do, people start to discuss “process”, “framework” etc. as if doing projects is like a manufacturing (stable) process.  Projects are nothing like this and trying to borrow concepts from the manufacturing world is only going to address a small part of delivering projects well. It can also lead to a major mismatch in terms of what is really needed.

Notes/ references:

  1. The following was sent to me in April 2018, involving a current project in a major US Government department – it is not an isolated case (the sender gave me permission to publish this): “Yep, we had lots of challenges implementing Agile for important parts of the project that didn’t involve software development — training, contract management, RFPs, HR, data migration, coordination with external integration partners, etc. Scrum seemed to pretend those things didn’t exist and no one needed to manage them.”. Under Sprint planning, the Guide says “The development team forecasts the functionality  that will be developed during the Sprint”. Clearly, there is a message here: Scrum teams work on functionality – they do not work on all those items listed above. “The Product Backlog is an ordered list of everything that is known to be needed in the product” – another statement in the guide that clearly defines the scope of Scrum.
  2. There is also the Scrum Master – but they are just a “master” of Scrum – they (usually) don’t do any work on the project. They are not going to manage the majority of what is in point #1 above, if any at all.
  3. Anything that includes both words “immutable“ and “rules” is prescriptive. There are many, many rules in Scrum. Prescriptive is anti-Agile. We will leave it to the readers to decide why Scrum is prescriptive.
  4. “https://ronjeffries.com/articles/2015-03-01-testing-in-sprints/” and “https://twitter.com/jeffsutherland/status/571006531103100929” and “https://www.scruminc.com/getting-to-done/” and “https://labs.openviewpartners.com/succeeding-with-scrum/#.WwrpCe4vzIU”
  5. “For the Product Owner to succeed, the entire organization must respect his or her decisions”.  That would be very nice – but will likely be a first. Is that what collaboration really is?  I must have been missing something all along.
  6. “Ensuring that the Product Backlog is visible, transparent, and clear to all, and shows what the Scrum team will work on next”. This is a full-time highly specialist role – in non-software houses especially, this would be beyond the capacity of most product owners. “The Product Owner tracks this total work remaining at least every Sprint Review. The Product Owner compares this amount with work remaining at previous Sprint Reviews to assess progress toward completing projected work by the desired time for the goal”. Really?
  7. “Each Sprint may be considered a project with no more than a one-month horizon”.
  8. Go to LinkedIn and put the word “Evangelist” into the search bar – look at how many profiles come up, and what they all have in common.

Share this:

  1. SithGotCore on August 7, 2019 at 7:16 pm

    Wow..this author really lacks the knowledge and experiences in implementing productive agile using SCRUM.

    • Kevin Lonergan on August 26, 2019 at 2:03 pm

      Hi Sith – thanks for your comment – perhaps you might like to comment more specifically on any of the items from 1-15 above and enlighten me on where my knowledge is lacking? For example, please look at item 1 and especially the notes referred to in item 1 and explain to me what I (and others) have missed or misunderstood? Lastly, Scrum, combined with the elements that Scrum chooses to deny the need for or the existence of, can work very well.

      • Nick Nijenhuis on May 24, 2020 at 10:13 pm

        The whole part you misunderstood is that Scrum is a framework and not a method.

        Completely agree with you that it won’t work without good other software practices. A lot of ‘agile coaches’ don’t even know what they are though.

        • Kevin Lonergan on May 25, 2020 at 12:01 pm

          Hi Nick – thanks for your comment. We seem to agree on the subject of Agile “coaches” and lack of software practices – however to me, an effective framework (and Agile) embraces flexibility for good reasons; Scrum, to me, is totally inflexible and full of “rules”. I would refer to this as a method and don’t pay much attention to what the owners of Scrum call it and I find much less value in debating whether it is a method or framework compared to addressing the specific issues I have raised in this post, which are supported by examples coming from the experiences of others in the real world.

          If you said Scrum is a (product development) method that sits within a more overall framework for delivering projects, that recognises and deals with the things that Scrum does not, then I might agree with you.

          We have another article that goes into this distinction much further: https://www.pmis-consulting.com/methods-methodologies-frameworks-and-bodies-of-knowledge-boks/

  2. malcolm west on January 16, 2020 at 4:17 pm

    Kevin,

    I agree. Many organisations don’t like Scrum because it doesn’t fit to what corporate managements often expect. So although we support many methods in our tools we had left Scrum in he too difficult pile and stuck with DSDM Atern for our agile offering. But having bitten the bullet about a year ago me took a similar approach to yours and we put the core Scrum in a planned wrapper. This included the sort of documentation and data collection that corporate management feels more comfortable with and allows the Scrum Project to be operated alongside and compared to other projects in the organisation.

    We called it Practical Scrum. It can be used for free forever in our Community Edition product.

  3. Kate Talbot on February 1, 2020 at 5:23 pm

    This is enlightening. A new manager introduced scrum/agile into a government IT department and things went to hell, technical debt built up, customers complained, teams put 400 tickets in the backlog and were ignored. A mess we are trying to clean up. This was a good read. Thank you

    • Kevin Lonergan on February 1, 2020 at 5:27 pm

      Hi Kate- many thanks for your comment. Your experience is sadly very common. The Scrum “zealots” usually leave a trail of failure in their wake.

  4. Natalia Melniciuc on May 25, 2020 at 3:53 pm

    Thank you for the article. It describes the reality in enterprises with Agile. Following the strict rules of Scrum ruined the main idea of IT project which includes multiple dependencies between different teams (including E2E tests, Business Acc Tests…). I consider, the main idea is to learn different methodologies and then project them on your realities – you can combine different ideas – these frameworks are just tools that should serve you, not you serve the tools.

    • Kevin Lonergan on May 25, 2020 at 4:22 pm

      Natalia – thank you for your excellent comments. You make some really important points. These are tools from which we can choose the best ones to help us – rules do not belong alongside the tools.

  5. Simon on November 18, 2020 at 7:46 am

    One question to be asked is: who controls the backlog? What gets measured gets managed, and in one company worked for this meant the software team having two backlogs – a public one (being measured) for new features and a private one (not being measured) for wishes, technical debt etc. We were highly discouraged from putting technical debt onto the public backlog…

    I compare this to another software house where ANYONE in the company could enter backlog items. (Admittedly, the receptionist never did: but there was no need to stop her doing so.) A backlog item could be anything from an actual bug to a wish, and the person submitting it didn’t need to find cause. We made it as easy as possible for anyone to enter these via a simple Web form. These were triaged very quickly (first thing each morning) by the team leader, but any developer could reassign them either to self or to others. (There wasn’t a problem with subterfuge here, but occasionally items would go back and forth between developers in good faith as they worked on a solution together.)

    This worked extremely well: we got very quick feedback, people were actually encouraged to submit things and knew they would be looked at, also they might not be acted on immediately. For this you need an experienced, strong team leader and a no-blame culture, both of which we had. We measured and rewarded people for ADDING items to the backlog, as well as for completing them.

    • Kevin Lonergan on November 18, 2020 at 9:14 am

      Hi Simon – many thanks for your comment. I wholeheartedly agree, transparency and strong leadership are always required in product development. Again, Scrum dances around the question of leadership saying things like “the team will decide” and denies the need for leadership. Sounds nice in principle but in the real world will fail too many times. I love the idea of inviting anyone to suggest ideas and then an efficient way to triage and implement them. I assume that for those that did not go ahead, the reason was made very clear and also published openly especially to the author of the suggestion.

      • simon on November 19, 2020 at 5:27 am

        Yes. issues that were “not for now” were retained in the system like all others, and reasons given in the free-form log, so anyone could see them. During project planning, those marked “not for now” would be reviewed to see whether they were to form part of the work for the next release. Sometimes you can kill a whole flock of birds with one stone this way. (But sometimes it’s quite a big stone.)

Leave a Comment





Contact Us Now

EMAIL us today.

Or call ++44 (0)1865 784040

Subscribe to Blog via Email

Enter your email to subscribe to our blog and get notifications of new posts by email. (We don't SPAM - ever.)

We use cookies to give you the best experience on our website. We do not store or collect any user data.