The secret to delivering software projects on-time and on-budget is to make an excellent effort of defining the requirements of the product and to be objective as to which requirements are absolutely necessary. Once the requirements are defined, they should be carefully reviewed at key junctures in the software development process to ensure that all the requirements are adequately defined and that each requirement is absolutely necessary for the completion of the Sprint, or more broadly speaking, the overall release of the software. Backlog Refinement is the key process within the Agile methodology that helps you achieve these goals.
In an earlier blog, we explored some best practices for documenting Agile product requirements. Product requirements are the common language between business and engineering that play a vital role because they are the first step in the product development process – where value creation is highest. In Agile projects, project requirements documentation should evolve, and stakeholders collaborate to keep documentation as light as possible, while providing the necessary information to design and complete a Sprint.
Requirements Scrubbing
Before the introduction of the Agile methodology, the term Requirements Scrubbing was used to describe what we now know as Backlog Refinement. Requirements scrubbing is the process of objectively reviewing each requirement in detail by the project team to determine whether the requirement is absolutely necessary for the upcoming release. We have since morphed the traditional time frame of a release to a Sprint, or series of Sprints. Any requirements that can be eliminated from the upcoming Sprint will have a direct savings on time and money, as it eliminates the need to go through design, development, testing and documentation. All requirements that are not absolutely necessary for the next Sprint can be designated for a future Sprint.
Each requirement should be evaluated for the following criteria:
- Is the requirement absolutely necessary for the upcoming Sprint? If not, it should be moved to a future Sprint.
- Is the requirement sufficiently defined so that the project team can easily provide a design for the requirement, specifically a User Story, a UI Design Interface, and associated integration points?
- Is the requirement in its simplest form? Often, requirements are not well conceived and can be more complex than is needed. This causes unneeded time and effort in the development and testing phases.
- Can other options be substituted for a requirement? There may be other options, particularly those already developed and tested, that yield the same result but with faster and improved execution.
Product Backlog Refinement
In Scrum parlance, the Product Backlog is an ordered list of all items that is known to be needed in the product. The earliest development of it provides the initially known and best-understood requirements. The Product Backlog evolves as the product and the environment in which it will be used evolves.
In turn, Product Backlog Refinement, or at times called Backlog Grooming, is the process through which Product Backlog items are reviewed by the project team and revised, providing more detail and ensuring that there is greater clarity in the requirements for that item. The Product Owner is responsible for the Product Backlog, including its content, detail, priority, and sequencing, but the project team including resident Product Managers and Business Analysts, play an important role. This important activity occurs at a strategic juncture in the Sprint and may be an officially scheduled meeting or an ongoing activity.
Often, Product Backlog Refinement occurs together with Sprint planning since they are intimately interrelated. Sprint Planning is a collaborative effort involving the Product Owner, a facilitator (often a Scrum Master) and the project team that clarifies the details of the product backlog items, defines their respective acceptance criteria, and defines the work and effort necessary to meet the Sprint commitment. The activity should answer two broad questions: What can be done in the Sprint? and How will the work get done?.
Let us now unveil the six tips to help you sharpen and fine tune your Backlog Refinement process.
Tip 1: Cross Team Participation
In order to carry out the refinement process effectively, it is critical to involve other Agile teams that have some level of dependency with your own team. It may not be possible to get all the members of the other teams in your refinement sessions, but at the very least you should include the product owners, scrum masters and architects from other teams to the extent possible. This will help you achieve a much wider visibility and help you uncover the dependencies and risk items that can otherwise be in a blind spot. This also helps you identify any missing user stories and technical debt.
Tip 2: Right Size Your User Stories
One of the key goals of the refinement process is to fix any deficiencies in the backlog vis-à-vis the size of the user stories. When user stories are first created, they are likely not sized appropriately and some need to be broken down into fine-grained user stories. There are many strategies and best practices around breaking down larger user stories. Take the perspective of the end-user and separate larger and ambiguous user stories into fine grained user stories that focus on one clear end user goal. The key criteria you should employ revolve around end user business functionality and testability of the user story. If the user story is too complex to be tested as a whole because it mixes multiple features, it may be a good candidate to be broken down further. A really large estimate for a user story may be another sign that it needs to be right sized.
Tip 3: Prioritize User Stories
Another important aspect of backlog refinement revolves around making sure the right user stories are at the top of the list. Prioritizing the user stories accurately needs to be done keeping the customer in mind. Which user stories are likely to bring the most benefit to the customer (as opposed to which user stories look most interesting to the engineering team)? This is a critical role for Product Manager or Product Owner to play as they are most intimately familiar with the customer needs. This can be done effectively when Product Managers work closely with the customer and track and follow what they desire from the product. Customer interactions from sales demos are also a great source of such information. Sales and marketing teams are a great source of such information because they are in touch with the markets and customers. If you can encourage them to participate in the refinement sessions, that could be very beneficial. This goes back to the issue of cross team participation we talk about earlier.
Tip 4: Estimate Like a Pro
Good estimation is very important—it helps product owners optimize resources for efficiency and better outcomes. In a previous blog, we explore this topic in depth. Accurately estimating user stories is fundamental to the success of the Agile process. Estimation needs to be fact based and data driven. Avoid the tendency to rely too much on “gut instinct”. Estimates are best done in a team setting with inputs from other team members and even from other teams. You should update user story estimates based on what you learn about hidden dependencies and risks, which you are likely to uncover in the refinement sessions. Some level of cross-checking and questioning is healthy in nailing down the estimates in order to reign in the “outlier” estimates. You also want to minimize large variations in estimates due to the fact that different people can estimate the same task with diverging estimates. You also should institutionalize the estimate variations based on the “skill level” of the resource (for example, a junior level person may take 30% more time for the same task).
Tip 5: Uncover Dependencies
A lot of the user stories depend on other user stories within the team and indeed on other agile teams as well. Dependencies can be of two types – hard and soft dependencies. Hard dependency means the user story cannot be started before the dependent user story is first completed. This leads to the issue of properly sequencing the user stories based on dependencies. This is a key goal of the backlog refinement process, and uncovering dependencies plays a key role in that. Soft dependencies may mean the user story must be at least done concurrently in the same sprint or in a later sprint. If dependencies are not properly identified, user stories may be sequenced incorrectly which leads to major re-planning work down the road when user stories get stuck in the development process for too long.
Tip 6: Identify Risks
Risk events are unforeseen occurrences that are difficult to predict in the first place, but they tend to derail progress in the most unexpected ways. There is always risk waiting to rear its ugly head when a user story gets underway in development. This could include risk from unexpected performance or scale issues, unexpected user experience issues or difficulty in testing from a myriad of reasons. A good way to identify these risks is to learn from the team’s past experience. Did similar user stories in the past run into any risks? It is good to maintain a running checklist of past risk items, and check them against the currently planned user stories. Once a potential risk is identified, the team should brainstorm and develop possible risk mitigation strategies for each risk. With a good risk mitigation plan in place, it is possible to mute the effect of many of these risk events.
The Role of Product Manager
The role of the Product Manager (or Business Analyst) in Backlog Refinement cannot be overstated. They are intimately involved in the original development of requirements for the product, often developing the overall direction and strategy from the stakeholders (Product Owner, Customer), fleshing out the strategy and breaking high level objectives into the more detailed business requirements. They must fully capture the requirements and make sure they are translated to understandable documentation for the full project team. Product Managers also have the big picture visibility that can help identify inter-dependencies and potential risks. Based on their prior experience, they can suggest risk mitigation strategies or a sequence of user stories that addresses the inter-dependencies.
Often, the Backlog Refinement process blurs the original intention of the business requirement and the Business Analyst must step in to ensure the intention and integrity of the original business objectives are being met. The most convenient juncture for this interaction is the review of the Product Requirements Document when business goals are relatively fresh in everyone’s mind. But, it becomes even more important as the project moves into several iterative Sprints and the tradeoffs between requirements, design and implementation are collaboratively discussed.
Closing Thoughts
When properly done, an effective Backlog Refinement process can dramatically increase your chances of delivering products on-time and on-budget. This effort does not stop with the review of the Product Requirements but is as active part of the Sprint Planning process. Product Managers and Business Analysts play a crucial role in this process and are integral to ensuring that the intention and integrity of the original business objectives are being met, while the trade-offs between requirements, design and implementation are collaboratively discussed. An efficient Backlog Refinement process plays a pivotal role in a timely delivery of product features that align with your overall business goals and customer needs. The six tips we present in this article will help you get there.
What are your experiences around Backlog Refinement? What best practices have you adopted and what lessons have you learned? We would love to hear from you in the comments section below.