Agile Planning and Estimation

Planning and Estimation are essential in software projects to achieve predictability, reduce the risks involved, and set a basic expectation for all stakeholders. Planning brings a lot of focus on preparation and forecasting whereas Estimation is a process to forecast project-related variables i.e., effort, scope, schedule, etc.  

Planning: Planning is required irrespective of the project management methodologies that the team follows, whether it is Waterfall or Agile. Planning gives the project team a perspective on how to meet the objective in a systematic way and helps project stakeholders to keep a tab on the project progress and investments done.  

As Mike Cohn defines it, “Agile planning balances the effort and investment in planning with the knowledge that we will revise the plan through the course of the project. An agile plan is one that we are not only willing but also eager to change”. This concept exists mainly to avoid the weakness of the planning.  

These weaknesses include:  

• Focusing on activities instead of delivered features
• Ignoring the prioritization
• Ignoring the existence of uncertainty
• Giving commitments based upon estimations

Such weaknesses in planning make the team not able to cope with the project dynamic environments. For resolving that, the planning must advance to become Agile as well.  

Estimation: Schedule, Scope, Cost, and Effort are the four major variables that typically control Software projects. Any changes in any of these variables can have an effect on a project. Estimation is a process to forecast these variables to develop or maintain software based on the information specified by the client.  

There are three main challenges faced during estimation i.e., Uncertainty, Self-knowledge, and Consistency of Method used for Estimation. Usage of standardized and scientific estimation methods for estimating size, effort, and schedule, helps towards maintaining minimal variance between the planned estimates and actual values thereby achieving maximum estimation accuracy. This provides a better client experience. All estimation needs for a project cannot be determined by a single method. It is important to have different methods of estimation for different stages.

Planning and Estimation in Agile projects bring a lot of focus on preparation and forecasting. Both these activities are done keeping business context in mind and measurable value delivery is committed to the client. Therefore, it is recommended to have required planning and estimation in Agile from the start of the project, in order to ensure better risk coverage and higher predictability.  

Planning and Estimation in Agile projects are generally done at three different stages, which are:  

• Project Initiation Level
• Release Level
• Sprint Level (i.e., an iteration within each Release)

Let us now understand each stage in detail.

Project Level:

Planning: 

Planning activities at the Project level takes 360 degrees of project view in terms of understanding the big picture, overall vision, roadmap, value delivery, planning for better risk coverage, creation of Product Backlog, defining the Releases, etc. This is the highest level of planning that focuses on the impending activities of the project and its ultimate goal. The Product Owner (or Client/Business manager) is the primary stakeholder involved in this activity.  

During this stage, the Product Owner (PO) explains to the project/delivery team the strategies which need to be followed by them to deliver the potentially shippable product on time. The team’s role here would be to understand the strategic plan clearly and fully fill the needs of the Product Owner to the fullest.  

Once the project goal is known, then it is required to define how to achieve that goal. It’s all about breaking down the requirements into various segments so that the final deliverables can be built, validated, and released. This helps to plan and estimate various releases as per the feature requirements. At this point, the Product Owner’s role becomes crucial. This planning leads to the generation of Product Backlog, which then is used in further planning i.e., Release Planning. Ignoring the Project Planning activity would lead to developing an end product that is not useful or not as expected by the stakeholders.  

In practice, planning at the Project level involves a lot of components that provide an overall control in order to achieve predictability in each delivery, mitigating the risks, and setting the basic expectation with all stakeholders. These components are related to infrastructure planning, quality management planning, environment setup planning, tools and reuse planning, build automation and continuous integration planning, etc. However, at a high level, the important aspects of planning include the creation of a Product Backlog, having Requirements Workshop, creation of a Release plan by defining the releases, and planning of Agile practices through Application Life Cycle Management tools, that need to be adopted in the project.  

• Product Backlog: Product Backlog is a prioritized list of simplified requirements which is necessary to accomplish the intended delivery of the product/feature. It is a key input for planning. Product Owners often prepare the backlog with the consultation of business strategists and end-users. The prioritization is often based on business value, release date, interdependency and is maintained in a commonplace such as centralized MS Excel file, ALM tools, etc.
• Workshops: The main purpose of conducting workshops is for gathering, understanding, and prioritize requirements, and defining a high-level plan for the Releases mainly during the Project Initiation phase. A workshop is an event in which a group of people collaborates to present various ideas in order to achieve a defined goal with the help of an impartial facilitator. It is not the same as brainstorming. In a workshop, there should be a clear idea of what is required and what should be the outcome.
• Application Life Cycle Management: Application Lifecycle Management (ALM) aids in managing Agile projects by providing mechanisms to deliver software more predictably and to drive business value. Agile ALM tools are used for planning and estimating user stories and sharing them within the team. It can be used for building a Product Backlog, Sprint Backlog, establishing team commitment and velocity, visualizing team activity and project progress via Burn Down charts, and reporting on team progress. Just by dragging up/down user stories in the backlog, the team can prioritize them in order. Also, teams can efficiently capture, self-assign, and manage their tasks using a good ALM tool.

Estimation:

During the Project Initiation stage, functional requirements are at a high level, or the details of requirements are not well-formulated and documented. In the Product Backlog, many requirements are still at the Epic or Feature level. At this point, estimation is required as it will help in the prioritization and planning of the Releases. Different estimation techniques can be used at this stage. Some of them are  

• QFPA (Quick Function Point Analysis)
• Use Case Points
• Complexity based estimation
• COCOMO (Constructive Cost Model)
• COSMIC FP

During the Project Initiation stage, estimation should be done as a group activity, where results should be compared and disagreements are resolved. All the project assumptions and risks should be highlighted clearly. Typically, the team for performing this activity will include:  

• Product Owner
• Clients & Business Stakeholders
• Project Manager
• Developers and Designers
• Testers

The general approach for estimation at the project level is illustrated by the diagram below:

The Project Level estimate will give details about the  

• The approximate number of Releases/Sprints that may be required before deployment
• Approximate end date of the project
• Approximate staffing plan
• Approximate effort plan etc.

The project-level estimates get refined regularly at the planning phase of each Release/Sprint.

Release Level:

Planning:

Without a release plan, the project team would be bouncing from one Sprint to the next, unsure of where to stop (i.e., logically). A shippable product is delivered and ready to be deployed at the end of each Release. As a result, it is critical for a project team to comprehend and adhere to the Release plan.

Release planning is all about planning and scheduling user stories (i.e., functional/non-functional requirements) that the team will turn into working software and deliver for that release. The team decides on the number of Sprints (duration 1 to 4 weeks) for each Release in a time-boxed manner where a set of user stories get converted into the working software while undertaking Release Planning.  

The release plan can also mention some key assumptions like how long a Sprint will be, how big the team will be, who will be working in the team, the starting of the first Sprint, the end of the last Sprint, etc. Releases are recommended to be planned anywhere between three to six months based on business context for the annuity (long duration i.e., longer than a year) projects, and for the non-annuity projects, one to three months.  

It is important to plan for the Releases, as it provides a clear picture to all the stakeholders, on what is expected to get into production, in due course of project execution. Release planning is an important element as it takes place before the start of each new release. This helps the team to deliver the expected project output on an incremental basis to the client which helps them to get early feedback so that future planning gets more aligned to the need of the client. 

All stakeholders identify and agree on the duration, goal, and content of each Sprint and Release. However, because requirements may be added or removed as the project advances, at the beginning of each Release and Sprint, the planning for the same is revisited and the scope is revised. The project team can decide to have additional Sprints or Releases during the course of the project depending on the dynamics at that point in time.  

Following are various activities involved in Release planning:

• Determine Objective of Release: The Product Owner will generally discuss the desired objective in the Release planning meeting. Mostly the objective will be either time\date driven output or functionality-driven output. When the target is time\date driven output, the Product Owner will expect the release to be finished by said date. Product Owners will usually set a target for completing ‘x’ functionalities by the end of the Release.
• Estimating the User Stories (Size): Product owners will always have a wish list of items that they would like to be targeted during every upcoming Release of the project. It is important to identify and estimate the size (complexity) of the User Stories based on the business value associated with them. These estimates provide key inputs that need to be considered for the Release planning to define the timelines for the Release as well as for the duration of the Sprint.
• Prioritizing User Stories: Projects always face a situation where there is either too much work to deliver or too less time to deliver. So, it is always advisable that the product owner prioritizes the user stories available, so that team can work on it accordingly based on the business necessity.
• Selecting stories and defining Release date: By this step, the agile team would have information about User Story estimates (in size/complexity), the prioritized user stories (based on business necessity), and the required duration for each Sprint. These inputs would help in Release planning which meets the objective as defined by the Product Owner (or Client) for a Release. All members of the project team, along with the Product Owner and any other stakeholders (i.e., end-users, etc.) are involved in this activity and the Release end date is defined collaboratively with a common consensus.

Estimation:

Release level estimation is done at the time of Release planning. The prioritized requirements are taken from Product Backlog which is in the form of User Stories. The User Stories are estimated in terms of Story Points during Release planning which focuses on estimating the size of the software to be delivered for that particular Release. The other inputs considered are project-level size, effort, cycle time, and available skills. Based on this estimation, a number of releases and a total number of story points in each release are planned for the overall project. 

Estimation during Release Planning is done by the entire team using one of the following techniques:  

• Wideband Delphi (or Consensus approach) follows the steps while performing the estimation of user story points using the Wideband Delphi method:
  • The coordinator discusses the User Stories with the team and the Product Owner.
  • Each team member estimates the number of person-days needed to complete each functionality.
  • The coordinator compares and analyses the individual estimations with the team.
  • The process is repeated until there is no significant difference in the estimation by the group.
  • This exercise can usually be done in three rounds.
• Estimation by Analogy (or Extrapolation) follows the steps while performing the estimation of user story points using the Estimation by Analogy method:
  • A user story’s estimation is based on its relationship with one or more references.
  • User stories of similar size are collected together and estimated as a group.
  • For example, a user story “Ability to add payee” is estimated to take 40 hours to complete. If the user story “Ability to edit payee” is half as complicated as “Add payee,” the effort required for this story will be 20 hours.
    Following is another depiction of an example:

• Planning Poker technique uses estimation cards, which are based on the Fibonacci series i.e. 0, 1, 2, 3, 5, 8, 13, 20, 40, 100, and ∞ while estimating the story points for the User Stories. Planning poker includes all the team members, i.e. developers, testers, analysts, etc. It’s a team-wide consensus-based relative sizing technique. Following are the steps followed while performing estimation using this method:
  • Each member/estimator has a deck of cards with a legitimate estimate on each card.
  • The moderator (who does not usually play) reads a story, which is then briefly discussed.
  • Any questions that are raised are answered by the Product Owner.
  • Each estimator chooses a card that represents their estimate and puts it facing down (hidden).
  • The cards are all turned over at the same time so that everyone can see them.
  • If the estimates differ, the highest and lowest estimators might be used to justify the difference.
  • Re-estimate until the story points are agreed upon.

Sprint Level:

Planning:

The release plan provides a high-level view of what the team intends to build, and what the team intends to deploy at the end of the Release. It does not provide a detailed view of how the team would plan to drive the work within Sprints. Once Releases and their Sprints are known, the team starts planning for each Sprint. This occurs at the beginning of each Sprint, i.e., on the first day of Sprint, and allows the team to be more explicit about what they will deliver at the end of the Sprint. The team considers the Scrum Master’s suggestions as well as the Product Owner’s prioritized list. The team then decides what they can take from the prioritized backlog for this Sprint, then breaks down the tasks for each user story and assigns it to themselves.

In the Sprint Planning meeting, the team will be focused more on what they need to do, to complete user stories selected for that Sprint. This meeting is attended by the Product Owner, Scrum Master, all the team members like analysts, programmers, testers, database engineers, user interaction designers, and so on. The output of the Sprint planning can be recorded in a simple spreadsheet or an ALM tool (Application Lifecycle Management), where the team will have all the user stories of that Sprint and its tasks mentioned sequentially (i.e. Sprint Backlog).

Team Capacity planning and user story commitment for a Sprint are always great challenges for a Scrum team. Before committing to a Sprint goal, it is important to know the current team capacity. Based on the Capacity of the Scrum team, the commitment would be done for the completion of a certain number of user stories (or User Story points) by the end of that Sprint. Based on these inputs and a consent agreement between the Product Owner, Scrum Team, and the Scrum Master, the tasks can be estimated exactly. In this way, the team capacity can be used for effective planning and estimation.

Sprint Planning is carried out at the outset of each Sprint. This helps the Team to bring focus on the Sprint goal. In Sprint Planning, the team decides what it will build in the upcoming Sprint and how they will build it. The team commits to the Sprint goal after breaking down user stories into tasks and doing task-level estimation. Sprint Planning is done by the Product Owner, Scrum Master, and the Team. During this meeting, team velocity, previous velocity trends (if available), and available team capacity is to be considered for effective planning.  

The Sprint Planning meeting is divided into two segments, each of which is allotted half of the Sprint Planning meeting’s time.  

Part 1: In the first part

• The team works with the Product Owner to determine what will be delivered as part of the Sprint.
• The Product Owner explains User Stories to the team.
• Details the interaction
• Reviews infrastructure and interface
• Explains the acceptance criteria

Part 2: In the second part

• The Team members decide how they are going to build this functionality into a product increment during the Sprint.
• The team selects the User Story
• User Story is broken into tasks
• Each of the tasks is estimated in hours
• The team agrees on the User Stories that will be done during the Sprint.

The time needed for the Sprint Planning meeting will be in similar proportion based on the duration of the Sprint.  

Estimation :

During Sprint Planning, prioritized stories from the product backlog are broken into tasks that can be estimated. In Sprint Planning, bottom-up estimation is used, and tasks are estimated in days/hours, resulting in the actual effort for the User Story.

Tasks can be defined related to design, coding, test scenarios, unit testing, test case execution, etc. Depending on the “Definition of Done,” the list of tasks may include Automation, Regression Testing, updating Style Guides, completing a code review, etc. The team has to understand that Sprint success means all the parameters are set as part of the Definition of Done.  

In Agile projects, ‘Definition of Done’ determines conditions to be considered for a User Story, and its associated tasks are completed, and ready for user acceptance. ‘Definition of Done’ provides a shared understanding among the project team members along with the Product Owner. When someone in the team says that a task is finished, then everybody in the team has a common understanding of what it means. Through ‘Definition of Done’, the team adheres to quality standards.  

Each team member is responsible for their own tasks, which they must estimate separately. On a daily basis, the task owner has to report the remaining (to-do) hours for the task, which is used to draw the Burn-down chart.  

It is asked many times if Tester should estimate for development effort also, and vice versa. All of the user story-related activities should be estimated together as a group. (i.e., testers and developers). Estimation is done based on relative sizing during Release Planning and is taken into account automatically. Each work should be estimated by its owner during Sprint planning.

As part of Sprint Planning, Scrum Master, Product Owner, and Project Team members start with prioritized Product Backlog to define the scope of the Sprint and create the Sprint Backlog. The Sprint Backlog consists of User Stories that would be accomplished during the current Sprint. The team together takes the User Stories from the Sprint Backlog and decomposes them into individual tasks that could be estimated.  

The ideal time to complete a task is estimated. Estimation of ideal time translates the size (measured in story points) of a task into a thorough estimate of effort. Typically, this effort is measured in days or hours. The task estimation is intended for the Sprint Backlog and is only present during the Sprint.

For example, as the team does the Sprint Planning, they will pick a User Story that may be sized to 5 Story Points. Then this User Story is further broken down into different working tasks, such as designing, detailing, implementing, and testing. The goal is to figure out how many User Stories the development team can commit to during a Sprint. The development team must be comfortable with the commitment and the Product Owners must be confident that the team will deliver on that commitment.  

Ideal Time Estimation is based on the Bottom-Up Approach where the business requirements are broken down into low-level activity by the team members and each activity is estimated separately. Team members are asked to sign up for tasks and estimate how long it will take them to complete them in hours or days.

When a team estimate using ideal time, they are referring to the time required for a programmer to complete a feature or task in comparison to other features or tasks. The team is aligned to accomplish the expected goal by displaying the effort remaining in a commonplace (i.e., Burndown Charts).

Individual team members select an activity and submit estimates for ideal time estimation of tasks. If there is a disagreement in these estimates among the team members, then they discuss it and come to a consensus.