PEARL IV : INVEST in good User Stories and SMART Tasks

PEARL IV : INVEST in good User Stories and SMART Tasks

In software development and product management, a user story is one or more sentences in the everyday or business language of the end user or user of a system that captures what a user does or needs to do as part of his or her job function. User stories are used with agile software development methodologies as the basis for defining the functions a business system must provide, and to facilitate requirements management. It captures the ‘who’, ‘what’ and ‘why’ of a requirement in a simple, concise way, often limited in detail by what can be hand-written on a small paper notecard.

User stories originated with Extreme Programming (XP), whose first written description in 1998 only claimed that customers defined project scope “with user stories, which are like use cases”. Rather than offered as a distinct practice, they were described as one of the “game pieces” used in the planning game. However, most of the further literature thrust around all the ways arguing that user stories are “unlike” use cases, in trying to answer in a more practical manner “how requirements are handled” in XP and more generally Agile projects. This drives the emergence, over the years, of a more sophisticated account of user stories.

In 2001, Ron Jeffries proposed the well-known Three C’s formula, i.e. Card, Conversation, Confirmation, to capture the components of a user story:

A Card (or often a Post-it note) is a physical token giving tangible and durable form to what would otherwise only be an abstraction;

A Conversation is taking place at different time and places during a project between the various stakeholders concerned by the given feature (customers, users, developers, testers, etc.), which is largely verbal but most often supplemented by documentation;

The Confirmation, the more formal the better, ensures that the objectives the conversation revolved around have been reached finally.

A pidgin language is a simplified language, usually used for trade, that allows people who can’t communicate in their native language to nonetheless work together. User stories act like this. We don’t expect customers or users to view the system the same way that programmers do; stories act as a pidgin language where both sides can agree enough to work together effectively.

User stories are written by or for business users or customers as a primary way to influence the functionality of the system being developed. User stories may also be written by developers to express non-functional requirements (security, performance, quality, etc.), though primarily it is the task of a product manager to ensure user stories are captured.

Agile projects, especially Scrum , use a product backlog, which is a prioritized list of the functionality to be developed in a product or service. Although product backlog items can be whatever the team desires, user stories have emerged as the best and most popular form of product backlog items.

While a product backlog can be thought of as a replacement for the requirements document of a traditional project, it is important to remember that the written part of an agile user story (“As a user, I want …”) is incomplete until the discussions about that story occur.

It’s often best to think of the written part as a pointer to the real requirement. User stories could point to a diagram depicting a workflow, a spreadsheet showing how to perform a calculation, or any other artifact the product owner or team desires.

When the time comes for creating user stories, one of the developers gets together with a customer representative, e.g. a product manager (or product owner in Scrum), which has the responsibility for formulating the user stories. The developer may use a series of questions to get the customer representative going, such as asking about the desirability of some particular functionality, but must take care not to dominate the idea-creation process.

UserStoryWorkshop

A user story is not a specification, but an communication and collaboration tool. Stories should never be handed off to the development team. They should rather be part of a conversation: The product owner and the team should discuss the stories, or even better, write them together. This leverages the creativity and the knowledge of the team and usually results in better user stories.

As the customer representative conceives a user story, it is written down on a note card (e.g. 3×5 inches or 8×13 cm) with a name and a brief description. If the developer and the customer representative find a user story deficient in some way (too large, complicated, or imprecise), it is rewritten until satisfactory – often using the INVEST guidelines. Commonly, user stories are not to be definite once they have been written down, since requirements tend to change throughout the development lifecycle, which agile processes handles by not carving them in stone upfront.

A Scrum epic is a large user story. There’s no magic threshold at which we call a particular story an epic. It just means “big user story.

“Theme” is a collection of user stories

Important considerations for writing user stories:

  1. Stakeholders write user stories. An important concept is that your project stakeholders write the user stories, not the developers. User stories are simple enough that people can learn to write them in a few minutes, so it makes sense that the domain experts (the stakeholders) write them. As its name suggests, a user story describes how a customer or a user employs the product. You should therefore tell the stories from the user’s perspective.
  2. Personas : A great way to capture your insights about the users and customers is to use personas. But there is more to it: The persona goals help you discover your stories. Simply ask yourself: What functionality does the product have to provide to meet the goal of the personas.A user persona is a representation of the goals and behavior of a hypothesized group of users. In most cases, personas are synthesized from data collected from interviews with users. They are captured in 1–2 page descriptions that include behavior patterns, goals, skills, attitudes, and environment, with a few fictional personal details to make the persona a realistic character. For each product, more than one persona is usually created, but one persona should always be the primary focus for the design.
  3. Use the simplest tool : User stories are often written on index cards as you see in Figure 2(below)  Index cards are very easy to work with and are therefore an inclusive modeling technique.
  4. Remember non-functional requirements: Stories can be used to describe a wide variety of requirements types. For example in Figure 1(below) the Students can purchase parking passes online user story is a usage requirement similar to a use case whereas the Transcripts will be available online via a standard browser is closer to a technical requirement.
  5. Indicate the estimated size: You can see in Figure 2 that it includes an estimate for the effort to implement the user story. One way to estimate is to assign user story points to each card, a relative indication of how long it will take a pair of programmers to implement the story. The team then knows that if it currently takes them on average 2.5 hours per point; therefore the user story in Figure 2 will take around 10 hours to implement.
  6. Indicate the priority: Requirements, including defects identified as part of your independent parallel testing activities or by your operations and support efforts, are prioritized by your project stakeholders (or representatives thereof such as product owners) and added to the stack in the appropriate place. You can easily maintain a stack of prioritized requirements by moving the cards around in the stack as appropriate. You can see that the user story card includes an indication of the priority; Agilists often use a scale of one to ten with one being the highest priority. Other prioritization approaches are possible – priorities of High/Medium/Low are often used instead of numbers and some people will even assign each card it’s own unique priority order number (e.g. 344, 345, …). You want to indicate the priority somehow in case you drop the deck of cards, or if you’re using more sophisticated electronic tooling. Pick a strategy that works well for your team. You also see that the priority changed at some point in the past, this is a normal thing, motivating the team to move the card to another point in the stack. The implication is that your prioritization strategy needs to support this sort of activity. The advice is to keep it simple.
  7. Optionally include a unique identifier: The card also includes a unique identifier for the user story, in this case 173. The only reason to do this would be to do this is if you need to maintain some sort of traceability between the user story and other artifacts, in particular acceptance tests.

Figure 1. Example user stories.

  • Students can purchase monthly parking passes online.
  • Parking passes can be paid via credit cards.
  • Parking passes can be paid via PayPal.
  • Professors can input student marks.
  • Students can obtain their current seminar schedule.
  • Students can order official transcripts.
  • Students can only enroll in seminars for which they have prerequisites.
  • Transcripts will be available online via a standard browser.

Figure 2. User story card (informal, high level).

Format

A team at Connextra developed the traditional user-story template in 2001:

“As a <role>, I want <goal/desire> so that <benefit>”

Mike Cohn, a well-known author on user stories, regards the “so that” clause as optional:

“As a <role>, I want <goal/desire>”

Chris Matts suggested that “hunting the value” was the first step in successfully delivering software, and proposed this alternative as part of Feature Injection:

“In order to <receive benefit> as a <role>, I want <goal/desire>”

Another template based on the Five Ws specifies:

“As <who> <when> <where>, I <what> because <why>.”

User Scenarios

A user scenario expands upon your user stories by including details about how a system might be interpreted, experienced, and used. Like user stories, you might imagine several scenarios for each persona group that you anticipate will make up your audience. Your scenarios should anticipate the user’s goal, specify any assumed knowledge, and speculate on the details of the user’s interaction experience.

User Stories and Planning

There are two areas where user stories affect the planning process on agile projects:

  1. Scheduling. In the agile change management management process where work items, including stories, are addressed in priority order. So, the implication is that the priority assigned to a story affects when the work will be done to implement that requirement. Project stakeholders are responsible for prioritizing requirements.  A numerical prioritization strategy is taken (perhaps on a scale of 1 to 20) and a MoSCoW (Must Should Could Won’t) approach is also used. Stakeholders also have the right to define new requirements, change their minds about existing requirements, and even reprioritize requirements as they see fit. However, stakeholders must also be responsible for making decisions and providing information in a timely manner.
  2. Estimating. Developers are responsible for estimating the effort required to implement the things which they will work on, including stories. The implication is that because you can only do so much work in an iteration, the size of the work items (including stories), affect when those work items will be addressed. Although you may fear that developers don’t have the requisite estimating skills, and this is often true at first, the fact is that it doesn’t take long for people to get pretty good at estimating when they know that they’re going to have to live up to those estimates. If you’ve adopted the pair programming practice then a user story must be able to be implemented by two people in a single iteration/sprint. Therefore if you’re working in one week iterations each user story must describe less than one week worth of work. Of course, if you aren’t taking a non-solo development approach such as pair programming the user story would need to be implementable by a single person within a single iteration. Large stories, sometimes called epics, would need to be broken up into smaller stories to meet this criteria.

INVEST

Bill Wake  introduced the INVEST mnemonic in his seminal post on creating better stories, suggesting they should be Independent, Negotiable, Valuable, Estimable, Small, and Testable. An acute observer of common disconnects between product people and developers, his mnemonic remains an excellent checklist for the creation of high quality product design inputs.

The acronym “INVEST” can remind you that good stories are:

I – Independent
N – Negotiable
V – Valuable
E – Estimable
S – Small
T – Testable

Independent

Ideally, you should be able to implement your stories in any order while still realizing the benefit of seeing something working at the end. This is important for two reasons. First, you lose a lot of the adaptability agile’s supposed to deliver if your stories have (many) interdependencies. Second, it makes it much more likely you’ll end up at the end of an iteration without working product that you can validate to make sure that you’re moving in a valuable direction.

Negotiable… and Negotiated

User stories are not requirements A good story is negotiable. It is not an explicit contract for features; rather, details will be co-created by the customer and programmer during development. A good story captures the essence, not the details. Over time, the card may acquire notes, test ideas, and so on, but we don’t need these to prioritize or schedule stories.They’re not another just another template to put on a piece of paper- they’re part of a process where the product person (original drafter) of the stories co-creates a working incarnation of the story with the implementer. This way, the original author gets someone to help them improve/”edit” their story and make sure the story’s interpreted into product in a way that delivers on something Valuable for the user.

Valuable

A story needs to be valuable. We don’t care about value to just anybody; it needs to be valuable to the customer. Developers may have (legitimate) concerns, but these framed in a way that makes the customer perceive them as important.

This is especially an issue when splitting stories. Think of a whole story as a multi-layer cake, e.g., a network layer, a persistence layer, a logic layer, and a presentation layer. When we split a story, we’re serving up only part of that cake. We want to give the customer the essence of the whole cake, and the best way is to slice vertically through the layers. Developers often have an inclination to work on only one layer at a time (and get it “right”); but a full database layer (for example) has little value to the customer if there’s no presentation layer.

Making each slice valuable to the customer supports XP’s pay-as-you-go attitude toward infrastructure.

Stories want to communication information. Don’t hide them on a network drive, the corporate intranet , or a licensed tool. Make them visible instead, for instance, by putting them up on the wall. A great tool to discover, visualize, and manage your stories is a Product Canvas pasted on the wall.

Estimable

A good story can be estimated. We don’t need an exact estimate, but just enough to help the customer rank and schedule the story’s implementation. Being estimable is partly a function of being negotiated, as it’s hard to estimate a story we don’t understand. It is also a function of size: bigger stories are harder to estimate. Finally, it’s a function of the team: what’s easy to estimate will vary depending on the team’s experience. (Sometimes a team may have to split a story into a (time-boxed) “spike” that will give the team enough information to make a decent estimate, and the rest of the story that will actually implement the desired feature.)

It should be possible for a developer with relevant experience to roughly estimate a story. That estimate’s then used by the product person to prioritize their list of stories, taking account of both value and cost (in terms of developer time).

Small

„ Large stories (epics) are

– hard to estimate
– hard to plan

They don’t fit well into single sprints
„ Compound story

– An epic that comprises multiple shorter stories

„ Complex story

– A story that is inherently large and cannot easily be  disaggregated into constituent stories

Compound stories sometimes hide assumptions, Split compound stories along operational boundaries (CRUD), Data Boundaries.

Good stories tend to be small. Stories typically represent at most a few person-weeks worth of work. (Some teams restrict them to a few person-days of work.) Above this size, and it seems to be too hard to know what’s in the story’s scope. Saying, “it would take me more than a month” often implicitly adds, “as I don’t understand what-all it would entail.” Smaller stories tend to get more accurate estimates.

Story descriptions can be small too (and putting them on an index card helps make that happen). Alistair Cockburn described the cards as tokens promising a future conversation. Remember, the details can be elaborated through conversations with the customer.

Detail can be added to user stories in two ways:

  • By splitting a user story into multiple, smaller user stories.
  • By adding “conditions of satisfaction.”

When a relatively large story is split into multiple, smaller agile user stories, it is natural to assume that detail has been added. After all, more has been written.

The conditions of satisfaction is simply a high-level acceptance test that will be true after the agile user story is complete

Epics are big, coarse-grained user stories. Starting with epics allows you to sketch the product functionality without committing to the details. This is particularly helpful for new products and new features: It allows you to capture the rough scope, and it buys you time to learn more about the users and how to best meet their needs. It also reduces the time and effort required to integrate new insights.

A story map is a graphical, two-dimensional product backlog. At the top of the map are big user stories, which are sometimes “epics” as Mike Cohn describes them, and other times correspond to “themes” or “activities”. These grouping units are created by orienting at the user’s workflow or “the order you’d explain the behavior of the system”. Vertically, below the epics, the actual story cards are allocated and ordered by priority. The first horizontal row is a “walking skeleton” and below that represents increasing sophistication.

In this way it becomes possible to describe even big systems without losing the big picture.

Testable

A good story is testable. Writing a story card carries an implicit promise: “I understand what I want well enough that I could write a test for it.” Several teams have reported that by requiring customer tests before implementing a story, the team is more productive. “Testability” has always been a characteristic of good requirements; actually writing the tests early helps us know whether this goal is met.

Strive for 90% Test Automation.

If a customer doesn’t know how to test something, this may indicate that the story isn’t clear enough, or that it doesn’t reflect something valuable to them, or that the customer just needs help in testing.

A team can treat non-functional requirements (such as performance and usability) as things that need to be tested. Figure out how to operationalize these tests will help the team learn the true needs.

For all these attributes, the feedback cycle of proposing, estimating, and implementing stories will help teach the team what it needs to know.

 Agile User Stories Prioritization 

Aim of Agile User Stories Prioritization  engineering actions are to contribute to business value that is described in terms of return-on investment of software product and it is very  essential for a software. For a product to be  successful, it is very important to identify the  correct equalizer among the competing quality  User Stories. From the customers’ view, the action  of continuous User Stories prioritization creates  the very core of today’s agile process

Continuous requirements prioritization  process from the customer‟s scope of vision forms the  essential of today‟s agile approaches. An essential  feature of any agile approach is an expressed focus on  making business value for the customer . Agile software process practitioners deem this approach  especially valuable for the software producers in a  circumstance that admits extremely uncertain requirements, experimentation with fresh  development technology, and customers willing to  explore the ways in which developing product can  assist their business goals.

The continuous  prioritization of requirements during the project acts  as fundamental role in accomplishing business value  creation. Requirements (re)prioritization at interiteration time is the means to align expert decisions to  the business strategy that aims the business value.

Creating a great user experience (UX) requires more than writing user stories. Also consider the user journeys and interactions, the visual design, and the nonfunctional properties of your product. This results in a holistic description that makes it easier to identify risks and assumptions, and it increases the chances of creating a product with the right user experience.

Agile Development SMART Tasks

A task communicates the need to do some work. Each team member can define tasks to represent the work that they need to accomplish. For example, a developer can define development tasks to implement user stories. A tester can define test tasks to assign the job of writing and running test cases. A task can also be used to signal regressions or to suggest that exploratory testing should be performed. Also, a team member can define a task to represent generic work for the project.

You link tasks to a user story to track the progress of work that has occurred to complete that story. After you define a task, you can link it to the user story that it implements

During the Iteration Planning meeting, stories and sometimes defects are broken down into tasks. The tasks are all the work the team must complete to develop, test and accept the story as done.  Tasks typically range in size from 1 hour to 2 days, depending on the length of the iteration. Two-week iterations ideally have tasks of 8 hours or less, and four week iterations have tasks sized at 2 days or less. Tasks larger than these guidelines should be broken down further to allow the team to incrementally complete the work and show progress. Keep tasks small. Estimate all tasks in hours. Iteration tracking will be done at story level to keep the team focused on business value.

The entire team works together to come up with the tasks for all the stories identified for the iteration. The majority of the tasks are determined during the iteration planning meeting, but in some cases the team may discover tasks as the work starts. It is appropriate to add and/or remove tasks on stories during the iteration.

When a task involves an external group – perhaps you want to demo your system to some project stakeholders, or deploy a testable version into your QA/test environment – then you should consider including the task as a reminder to coordinate with those groups.

Tasks may include the traditional steps in a development lifecycle (although limited to the feature in question, not the entire product). For instance: Design, Development, Unit Testing, System Testing, UAT (User Acceptance Testing), Documentation, etc. Remember, agile software development methods do not exclude these steps. Agile methods just advocate doing the steps feature-by-feature, just in time, instead of in big phases.

At the end of the day , the individuals update the tasks hours remaining for the tasks they have taken responsibility . This will enable to determine the Burn down chart.

Tasks makes the micro work break down structure that the teams can use to facilitate coordinating, estimating, tracking status and and assigning individual responsibilities to help assure completion of the stories. and there by the iteration

There is an acronym for creating effective goals: “SMART” –

S – Specific
M – Measurable
A – Achievable
R – Relevant
T – Time-boxed
These are good characteristics for tasks as well.

Specific

A task needs to be specific enough that everyone can understand what’s involved in it. This helps keep other tasks from overlapping, and helps people understand whether the tasks add up to the full story.

Measurable

The key measure is, “can we mark it as done?” The team needs to agree on what that means, but it should include “does what it is intended to,” “tests are included,” and “the code has been refactored.”

Achievable

The task owner should expect to be able to achieve a task. XP teams have a rule that anybody can ask for help whenever they need it; this certainly includes ensuring that task owners are up to the job.

Relevant

Every task should be relevant, contributing to the story at hand. Stories are broken into tasks for the benefit of developers, but a customer should still be able to expect that every task can be explained and justified.

Time-Boxed

A task should be time-boxed: limited to a specific duration. This doesn’t need to be a formal estimate in hours or days, but there should be an expectation so people know when they should seek help. If a task is harder than expected, the team needs to know it must split the task, change players, or do something to help the task (and story) get done.

As you discuss stories, write cards, and split stories, the INVEST acronym can help remind you of characteristics of good stories. When creating a task plan, applying the SMART acronym can improve your tasks.

Scrum and Sprints

The task board is the single most important information radiator that an agile team has. A task board illustrates the progress that an agile team is making in achieving their sprint goals. Usually the task board is located in an area that is central to the team

All of these task boards have a few simple things in common:

1) 4 basic columns (there can be more)

  1. Stories
  2. To Do
  3. In Process (In progress, WIP, etc.)
  4. Complete (Completed, Done, etc.)

2) A Row for each story

Simple task board

Taskboard

More complicated version that looks something like above.

PEARL I : SCRUM : An Introduction on Origin,Evolution,Definition,Adoption and Future

PEARL I : SCRUM : An Introduction on Origin,Evolution,Definition,Adoption and Future

Origin

The first software development Scrum was created at Easel Corporation  in 1993 based on extensive research on successful projects worldwide, a deep analysis of the computer science literature, close collaboration with leading productivity experts, and decades of experience with advanced software technologies. Jeff Sutherland was the Chief Engineer for the Object Studio team that defined roles, hired the first Product Owner and ScrumMaster, developed the first Product Backlog and Sprint Backlog and built the first portfolio of products created with Scrum. They built the first object-oriented design and analysis tool that  incorporated round-trip engineering in the initial SCRUM. A second SCRUM  implemented the first product to completely automate object-relational mapping in an  enterprise development environment. Jeff was assisted by two world-class developers, Jeff  McKenna, now an extreme programming (XP) consultant, and John Scumniotales, now a  development leader for object-oriented design tools at Rational Corporation.

There were some key factors that influenced the introduction of SCRUM at Easel Corporation. The book “Wicked Problems, Righteous Solutions” (DeGrace and Stahl 1990) reviewed the reasons why the waterfall approach to software development does not work for software development today. Requirements are not fully understood before the  project begins. The user knows what they want only after they see an initial version of the  software. Requirements change during the software construction process. And new tools  and technologies make implementation strategies unpredictable. DeGrace and Stahl  reviewed “All-at-Once” models of software development which uniquely fit object oriented implementation of software.
In 1995, Jeff introduced the Scrum team to Ken Schwaber, CEO of Advanced Development Methods. Ken agreed that Scrum was a better way to build software and they worked together to formalize the Scrum development process at OOPSLA’95 . In the same year, Sutherland provided support for development of eXtreme Programming by giving Kent Beck all background information on the creation of Scrum and the results of two years of product development with the Scrum process from 1993-95. XP engineering practices evolved along with Scrum and the two leading Agile development processes work well together. Scrum and XP are the most widely used Agile processes worldwide and their creators are signatories of the Agile Manifesto.

One of the influences that sparked the creation of the Scrum Agile development process was a Harvard Business Review paper on Japanese new product  development by Takeuchi and Nonaka  . A key  component of their presentation was a chart showing product development separated into silo’s (Type A),  phases slightly overlapped (Type B), and all phases of  development overlapping (Type C). The Authors viewed  Type A product development as implemented at NASA as  an outmoded relay race process. Fuji-Xerox abandoned the  NASA approach for Type B which they called “Sashimi”  because slices of work overlapped with collaboration  between phases. Type C was implemented at Canon and  Honda. Takeuchi and Nonaka envisioned Types B and C as  a Rugby approach where multiple phases of product  development are done simultaneously. Scrum is a Rugby  formation and they viewed an “all-at-once” process as  similar to a Rugby team moving down the field passing the  ball back and forth

Jeff Sutherland got the name “Scrum” from the famous article “The New New Product Development Game,” which was published by Takeuchi and Nonaka in the Harvard Business Review in 1986. This article describes how companies like Honda, Cannon, Fuji-Xerox produced world class results using a scalable, team based approach to all at once product development. It also emphasizes the importance of empowered self organizing teams and outlines managements role in the development process.

On the first page of the article they write: “Companies are increasingly realizing that the old, sequential approach to developing new products simply won’t get the job done. Instead companies

in Japan and the United States are using a holistic method – as in rugby, the ball gets passed within the team as it moves as a unit up the field.

The Evolution

Evolution occurs in dynamic response to environmental demands.

After discussing the notion of various types of Scrum  with Certified ScrumMasters in Scrum Alliance workshops  and with development teams at Microsoft, Yahoo, Ariba,  Adobe, GE Healthcare, and other companies, it appeared  that the chart in article can be applied to a higher level of  thinking about three types of Scrum—going beyond the  thinking of Takeuchi and Nonaka

In a Type A Scrum, all development occurs in an increment within the time box of Scrum iteration called a Sprint. A side effect of this approach is downtime between iterations when reorganizing for the next Sprint. Nevertheless, well executed Sprints can double productivity and repeatedly deliver projects on time, within budget, with functionality precisely targeted to end-user demands.
By adding product definition tasks for the next Sprint into the current Sprint, a Type B Sprint allows work to flow smoothly from Sprint to Sprint. Product backlog requirements for the next Sprint are developed in the current Sprint. This has enabled some development organizations to deliver more working product than sales, marketing, or customers can absorb. The development bottleneck is eliminated and the company can adopt new strategies and create new products that were previously impossible to deliver.
Type C Sprint can be envisioned as overlapping Sprints by running software releases through the same Scrum team at the same time. This requires experienced Scrum teams, well designed product architecture, and automation of Product and Sprint backlogs. Throughput can be enhanced to deliver dozens of new releases of enterprise software annually. Competitors can be overwhelmed and market dominance achieved.

Type C Scrum increases speed of development,  aligns individual and corporate objectives, creates a  culture driven by performance, supports shareholder  value creation, achieves stable and consistent  communication of performance at all levels, and enhances  individual development and quality of life

To deal with simultaneous weekly and monthly  Sprints, along with quarterly major releases, a MetaScrum  was formed at PatientKeeper led by the lead Product  Owner. This is a weekly meeting that typical takes 1.5  hours and includes the CEO and other senior  management, as well as leadership from marketing,  development, quality assurance, installation, and support.
Takeuchi and Nonaka observed that collapsing phases of product development improved innovation, throughput, time to market, and product acceptance. As market pressures have evolved and changed, it is possible to collapse Scrum Sprints to create a dramatic increase in business opportunity.

Sutherland and Schwaber evolved  Scrum through trial and error at a number of companies during the  s – NewsPage, IDX Systems, Fidelity Investments, and PatientKeeper.  The Daily Scrum, the burndowns and  the Sprint Backlog were added, and  planning Sprints were dropped. !e  importance of self-managing, cross functional teams became apparent and a bedrock to Scrum.

Schwaber also collaborated with process control scientists at DuPont to tie Scrum to first principles in industrial process control theory, learning that the reason Scrum worked was
that it was empirical.

“In , the Agile Manifesto was formed,” says Schwaber. “!e signatories gathered in SnowBird because they felt their approach to software development had  something in common that was far better and more people-oriented than the  emerging favored process of the day, Rational Unified Process. !e principles  of the manifesto are the common touch
points that we agreed upon.”
Today, Schwaber notices that the term “Agile” is often used in a careless way.“Many people think that there are Agile processes. There aren’t. There are processes that conform to the Agile  Manifesto principles, such as Scrum and  Extreme Programming. However, the  word Agile is often used as a description  for ‘we aren’t using waterfall.’ That is not what Agile means!”
In Schwaber’s mind, Scrum and Agile  started emerging for two reasons: The Agile Manifesto was a rallying point for dissatisfaction and a new  approach. The emergence of new IDEs  that were as powerful as SmallTalk, like VisualStudio and Eclipse, were the technological enablers.

Scrum has streamlined software development — and professionals from around the world are starting to see the value of using Scrum. Of all the possible Agile frameworks used by companies, 66 percent are Scrum or Scrum variants.

Scrum has the power to transform project management across every industry, every business, and even across life in general. By using Scrum, teams become more Agile, discovering how to react more quickly and respond more accurately to the inevitable change that comes in the teams way. And by staying focused, collaborating, and communicating, you can accomplish what truly needs to be done — successfully.

Most important, Scrum is not unproven hype. It’s a solid and successful Agile framework that’s been applied to a variety of projects and teams. Universities use Scrum to deliver valued projects to clients. Militaries have relied on Scrum to prepare ships for deployment. In the automotive world, Team Wikispeed is using Scrum to build a fast, affordable, ultra-efficient, safe commuter car that should sell for less than $20,000.

So whether you’re working on the next smartphone app, managing logistics for a store, or planning a charity event, you should take a closer look at using Scrum. And Scrum Alliance can give you the proven framework, best implementation practices, and supportive guidance you need to achieve success.

The most profitable software product ever created (Google Adwords ) is powered by Scrum and the most productive large project with over a million lines of code (SirsiDynix and Starsoft) used a distributed, outsourced, Scrum implementation. CMMI Level 5 companies cut costs in half with Scrum while simultaneously improving quality, customer satisfaction, and the developer experience (Systematic Software Engineering). At the same time, Scrum remains the process of choice in small entrepreneurial companies where it has it roots.

Scrum Theory
Scrum is founded on empirical process control theory, or empiricism. Empiricism asserts that  knowledge comes from experience and making decisions based on what is known. Scrum employs an iterative, incremental approach to optimize predictability and control risk.  Three pillars uphold every implementation of empirical process control: transparency,  inspection, and adaptation.

Transparency
Significant aspects of the process must be visible to those responsible for the outcome.
Transparency requires those aspects be defined by a common standard so observers share a  common understanding of what is being seen.
For example:

  • A common language referring to the process must be shared by all participants; and,
  • Those performing the work and those accepting the work product must share a common definition of “Done”

Inspection
Scrum users must frequently inspect Scrum artifacts and progress toward a Sprint Goal to detect  undesirable variances. Their inspection should not be so frequent that inspection gets in the way  of the work. Inspections are most beneficial when diligently performed by skilled inspectors at  the point of work.
Adaptation
If an inspector determines that one or more aspects of a process deviate outside acceptable  limits, and that the resulting product will be unacceptable, the process or the material being  processed must be adjusted. An adjustment must be made as soon as possible to minimize  further deviation.
Scrum prescribes four formal events for inspection and adaptation, as described in the Scrum

  •  Sprint Planning
  •  Daily Scrum
  •  Sprint Review
  •  Sprint Retrospective

Definition

Scrum is based on small set of values, principles, and practices called collectively the Scrum Framework. Organizations using scrum shall use the scrum framework entirely perhaps not through the entire organization at once, but certainly within the initial teams that use scrum

Core Scrum — Values and roles

Values from the Agile Manifesto

Scrum is the best known of the Agile frameworks. It is the source of much of the thinking behind the values and principles of the Agile Manifesto, which forms a common basis for all of these approaches.

The Agile Manifesto values apply directly to Scrum:

  • Individuals and interactions over processes and tools. Scrum, like all the Agile frameworks and methods, relies directly on trust in teams, the individuals in the teams, and the way they interact. Teams figure out what is to be done, teams figure out how to do it, and teams do it. Teams identify what’s getting in their way, and they take the responsibility to resolve all the difficulties that are within their scope. Teams work with other parts of the organization to resolve the concerns that are outside their control. This is critical. Trying to do Scrum but undermining this primary focus on team responsibility and autonomy will generally lead to trouble.
  • Working software over comprehensive documentation. Scrum requires a working, finished increment of the product as the primary result of every sprint. Certainly there will be analysis work, design work, and testing work, all of which may need to be documented. But it is having a result (working software) that allows the organization to guide the project to success. This is critical. Scrum teams must produce a product increment in every sprint.
  • Customer collaboration over contract negotiation. The Scrum product owner is the Scrum team’s prime point of contact with the eventual end users of the product. The product owner is a member of the team and works collaboratively with the team to determine what needs to be done. In this collaboration, the product owner selects the most valuable next things to do, ensuring that the product has the highest possible value at every point in time. This is critical. The product owner needs to build a rich collaboration with the team.
  • Responding to change over following a plan. Everything about Scrum is designed to make sure that everyone has the information they need to make good decisions about the project. The project’s progress is represented by a real, running product increment. The backlog of things to be done is available for all to see. Progress, both overall and sprint by sprint, is clearly visible. Problems and concerns are discussed openly and dealt with immediately. This is critical. Scrum works well for teams that openly inspect what’s going on and adapt their actions to the reality. It works poorly for those who do not.

Scrum values

All work performed in Scrum needs a firm foundation of values for the team’s process and principles. With its emphasis on teamwork and continuous improvement, Scrum both creates those values and relies on them. The values are focus, courage, openness, commitment, and respect. From the team’s perspective, the values offer the following guides:

  • Focus. Because we focus on only a few things at a time, we work well together and produce excellent work. We deliver valuable items sooner.
  • Courage. Because we are not alone, we feel supported and have more resources at our disposal. This gives us the courage to undertake greater challenges.
  • Openness. As we work together, we practice expressing how we’re doing and what’s in our way. We learn that it is good to express concerns so that they can be addressed.
  • Commitment. Because we have great control over our own destiny, we become more committed to success.
  • Respect. As we work together, sharing successes and failures, we come to respect each other and to help each other become worthy of respect.

If an organization will let Scrum do its work, everyone involved will discover the benefits and will begin to understand why Scrum both engenders and relies upon these values.

The Scrum framework

Scrum is a framework for building a product.

Scrum is also a team process that begins when stakeholders need a product. The Scrum team includes three roles: the product owner, the ScrumMaster, and the members of the development team. These roles are further described below. The product is built incrementally over a series of short time periods called sprints. A sprint is a fixed time period, up to four weeks long but with a preference toward shorter intervals. During each sprint, the Scrum team builds and delivers a product increment. Each increment is a recognizable, visibly improved, operating subset of the product, meeting understood acceptance criteria and built to a level of quality referred to as the Definition of Done.

Scrum includes three essential artifacts: the product backlog, the sprint backlog, and the product increment. The product backlog is the list of ideas for the product, in the order we expect to build them. The sprint backlog is the detailed plan for development during the next sprint. The product increment, a required result of every sprint, is an integrated version of the product, kept at high enough quality to be shippable if the product owner chooses to ship it.

In addition, Scrum requires transparency within the team and with the stakeholders. Therefore the Scrum team produces visible displays of plans and progress.

Scrum also includes five activities or meetings. These are the backlog refinement, sprint planning, daily scrum, sprint review, and sprint retrospective.

SCRUM project planning uses lightweight techniques such as Burndown  Charts, as opposed to PERT charts. A PERT chart is only as good as the assumptions  inherent in the critical path represented on the chart. In agile development, the critical  path usually changes daily, rendering any given PERT chart obsolete within 24 hours.  The solution is using a technique to calculate the velocity of development. The neural  networks in the brains of team members are used on a daily basis to calculate the critical  path. This allows the plan to be recalculated and the velocity of “burndown” of work to  be computed. Team efforts to accelerate or decelerate the velocity of burndown allow a  team to “fly” the project into a fixed delivery date.

SCRUM is  the only agile methodology that has been formalized and published as an organizational  pattern for software development (Beedle, Devos et al. 1999). The process assumes that  requirements will change during the period between initial specification and delivery of a  product. It supports Humphrey’s Requirements Uncertainty Principle which states that for  a new software system, the requirements will not be completely known until after the  users have used it. SCRUM allows for Ziv’s Uncertainty Principle in software  engineering – uncertainty is inherent and inevitable in software development processes  and products (Ziv and Richardson 1997). And it support Wegner’s Lemma which states  that it is not possible to completely specify an interactive system (Wegner 1995). Most  software systems built today are object-oriented implementations which depend on  environmental inputs to determine process outputs, i.e. interactive systems.

Scrum roles

Product owner

The product owner has responsibility for deciding what work will be done. This is the single individual who is responsible for bringing forward the most valuable product possible by the desired date. The product owner does this by managing the flow of work to the team, selecting and refining items from the product backlog. The product owner maintains the product backlog and ensures that everyone knows what is on it and what the priorities are. The product owner may be supported by other individuals but must be a single person.

Certainly the product owner is not solely responsible for everything. The entire Scrum team is responsible for being as productive as possible, for improving its practices, for asking the right questions, for helping the product owner.

Nonetheless, the product owner, in Scrum, is in a unique position. The product owner is typically the individual closest to the “business side” of the project. The product owner is charged by the organization to “get this product out” and is the person who is expected to do the best possible job of satisfying all the stakeholders. The product owner does this by managing the product backlog and by ensuring that the backlog, and progress against it, is kept visible.

The product owner, by choosing what the development team should do next and what to defer, makes the scope-versus-schedule decisions that should lead to the best possible product.

Development team member

The development team is made up of the professionals who do the work of delivering the product increment. Scrum requires that the development team be a cross-functional group of people who, as a group, have all the necessary skills to deliver each increment of the product.

The development team members are expected to be available to the project full time, not splitting their time over numerous projects. They have the responsibility of self-organizing to accomplish each sprint goal, producing each new product increment according to each sprint plan.

The product owner makes an ordered list of what needs to be done; the development team members forecast how much they can do in one sprint, and they decide how they are going to do it.

ScrumMaster

The ScrumMaster is a “servant leader” who helps the rest of the Scrum team follow the process. The ScrumMaster must have a good understanding of the Scrum framework and the ability to train others in its subtleties.

The ScrumMaster helps the product owner understand how to create and maintain the product backlog. He or she works with the entire Scrum team to evolve the Definition of Done. The ScrumMaster also works with the development team to find and implement the technical practices needed to get to Done at the end of each sprint.

Another responsibility of the ScrumMaster is to remove impediments to the team’s progress. These impediments may be external to the team (such as a lack of support from another team) or internal (such as the product owner not knowing how to properly prepare the product backlog). That said, the ScrumMaster fosters self-organization, meaning that the team itself should remove issues wherever possible.

The ScrumMaster may facilitate meetings and always acts as a coach for the Scrum team, helping it execute the Scrum process. He or she helps team members work together and learn the Scrum framework, and protects them from both internal and external distractions. The ScrumMaster keeps the Scrum team on track, productive, and growing in ability.

Ultimately, the ScrumMaster is responsible for ensuring that Scrum is understood and in place, inside the team and outside. He or she helps people outside the team understand the process and the kinds of interactions with the team that are helpful (and those that are not). The ScrumMaster helps everyone improve to make the Scrum team more productive and valuable.

Artifact: Product backlog

The product backlog is crucial tool that enables the scrum team to achieve fast, flexible, value delivery flow in the presence of uncertainty.

The product backlog is an essential artifact in Scrum. It is an ordered list of ideas for the product, kept in the order in which the team expects to do them. It is the single source from which all requirements flow. This means that all the work the development team does comes from the product backlog. Every feature idea, enhancement, bug fix, documentation requirement — every bit of work the team does — is derived from a product backlog item. Each item on the product backlog includes a description and an estimate of the team capacity required to produce it.

Scrum Overview

The product backlog may begin as a large list or a short one. It may be vague or detailed. Typically, it begins short and vague and becomes longer and more concrete as time goes on. Product backlog items slated for implementation soon will be clarified, better defined, and split into smaller chunks as part of the product backlog refinement activity.

The Product Owner is accountable for maintaining the product backlog, although he or she may — and should — have help in producing it and keeping it up to date. Individual product backlog items may originate from the Product Owner, from team members, or from other stakeholders.

Activity: Product backlog refinement

Since product backlog items are often large and general in nature, and since ideas come and go and priorities change, product backlog refinement is an ongoing activity throughout a Scrum project. This activity includes but is not limited to:

  • Keeping the product backlog ordered
  • Removing or demoting items that no longer seem important
  • Adding or promoting items that arise or become more important
  • Splitting items into smaller items
  • Merging items into larger items
  • Estimating items

One key benefit of the product backlog refinement activity is that it helps prepare for upcoming sprints. Considerations include but are not limited to:

  • Whether each item entering the sprint truly represents an increment of “business value”
  • Whether the development team needs to be able to build each item within just a single sprint
  • Whether everyone on the team has a clear understanding of the purpose of each item

Depending on the nature of the product, other skills and inputs may be necessary. In every case, product backlog refinement is best considered as an activity for all team members, not just for the Product Owner.

Activity: Sprint planning

Each sprint begins with a time-boxed meeting called the sprint planning meeting. In this meeting the Scrum team collaborates to select and understand the work to be done in the upcoming sprint.

The entire team attends the sprint planning meeting. Working from the ordered product backlog, the Product Owner and the development team members discuss each item and come to a shared understanding of that item and what is required to complete it according to the current Definition of Done. All Scrum meetings are time-boxed. The recommended time for the sprint planning meeting is two hours or less per week of sprint duration. Because the meeting is time-boxed, the success of the sprint planning meeting is highly dependent upon the quality of the product backlog going in. This is why product backlog refinement is an important Scrum activity.

Scrum moves control from a central scheduling dispatching authority to individual teams doing the work.

In Scrum, the sprint planning meeting is described as having two parts:

  1. Determine what work will be completed in the sprint
  2. Determine how the work will be accomplished

In the first part of the meeting, the Product Owner presents ordered product backlog items to the development team, and the whole Scrum team collaborates to understand the work.

The number of product backlog items to undertake in the sprint is solely up to the development team. To decide how many items to undertake, the development team considers the current state of the product increment, the past performance of the team, the team’s current capacity, and the ordered product backlog. The development team alone decides how much work to take on. Neither the Product Owner nor any other agency can push more work onto the development team.

Often, but not always, the sprint has a goal. Having a sprint goal is a strong practice that helps everyone focus more on the essence of what needs to be done and less on small details that may not truly be important for the final product.

In the second part of the meeting, the development team collaborates to decide how to produce the next product increment in accordance with the current Definition of Done. The team does sufficient design and planning to be confident of completing the work during the sprint. Work to be done in the early days is broken down into small units of one day or less. Work to be done later may be left in larger units to be decomposed later.

Deciding how to do the work is the responsibility of the development team, just as deciding what to do is the responsibility of the Product Owner.

The Product Owner needs to be readily available to the team even if he or she does not attend this part of the meeting. However, the Product Owner may remain to answer questions and resolve misunderstandings.

Sprint planning concludes with the Scrum team reaching a common understanding of the quantity and complexity of what is to be accomplished during the sprint and with a rational expectation of being able to complete it. The development team forecasts the amount of work it will complete and commits to accomplishing it.

To summarize, in the sprint planning meeting the development team:

  • Considers and discusses product backlog items with the Product Owner
  • Ensures that all team members understand those items
  • Selects a number of items to achieve
  • Creates a sufficiently detailed plan to ensure that achievement

The resulting list of things to do is the sprint backlog.

Artifact: Sprint backlog

The sprint backlog is the list of refined product backlog items chosen for development in the current sprint, together with the team’s plan for accomplishing that work. It reflects the team’s forecast of what work can be completed.

With the sprint backlog in place, the sprint begins, and the development team develops the new product increment defined by the sprint backlog.

Development

During the sprint, the development team self-organizes to produce the defined product increment. Self-organizing means that the development team determines how to produce and then does produce the product increment, in accordance with organizational standards, according to the Definition of Done.

Artifact: Product increment

The most important Scrum artifact is the product increment. Every sprint produces a product increment that must be of high enough quality to be given to users. The product increment must meet the Scrum team’s current Definition of Done, and each component of it must be acceptable to the Product Owner.

Additional indicators of visible progress

Scrum requires transparency within the team and outside the team. While the product increment is the most important way of creating transparency, the Scrum team will create any other artifacts that are needed to make sure that the status of the project is understood. Such artifacts commonly include burn charts and task boards.

Definition of Done

When the product increment is delivered, it needs to be “done” according to a shared understanding of what “done” means. This definition is different for every Scrum team, and as the team matures the Definition of Done will expand and become more stringent.

The Definition of Done must always include the notion that the product increment be of high enough quality to be shippable: The Product Owner could choose to release it immediately. The product increment includes the functionality of all previous product increments and is fully tested so that all completed product backlog items continue to work together.

Activity: Daily Scrum

The self-organizing development team uses the daily Scrum meeting to ensure that it is on track for attaining the sprint goal. The meeting takes place at the same time and place, every day. Each development team member gives three pieces of information:

  • What I have accomplished since our last daily Scrum
  • What I plan to accomplish between now and our next daily Scrum
  • What (if anything) is impeding my progress

There may be brief, clarifying questions and answers, but there is no discussion of any of these topics during the daily Scrum itself. However, many teams meet right after the daily Scrum to work on any issues that have come up.

The daily Scrum is not a report to management, nor to the Product Owner, nor to the ScrumMaster. It is a communication meeting within the development team that ensures that team members are all on the same page. Only the Scrum team members, including ScrumMaster and Product Owner, speak during this meeting (although other interested parties may come and listen). Based on what comes up in the meeting, the development team reorganizes the work as needed to accomplish the sprint goal.

The daily Scrum is a key element of Scrum, leading to transparency, trust, and better performance. It provides rapid recognition of problems and builds the team’s self-organization and self-reliance. All Scrum meetings are timeboxed; the recommended timebox for the daily Scrum is no more than 15 minutes.

Activity: Sprint review

At the end of the sprint, the Scrum team and stakeholders review the sprint’s output. The recommended timebox for the sprint review is one hour per week of sprint duration.

The central point of discussion is the product increment completed during the sprint. It is generally both wise and helpful for the stakeholders to attend this meeting, since they are directly interested in the product. This is an informal meeting to look at where the team is and to collaborate on how it might go forward. Everyone has input at the sprint review. Naturally, the Product Owner makes the final decisions about what happens next, updating the product backlog as appropriate.

Every teams will find its own way to handle the sprint review. A demonstration of the product increment is a common procedure. Groups also often discuss what they observed during the sprint and what product ideas came to mind. They discuss the state of the product backlog and talk about possible completion dates and what might be achieved by those dates.

The sprint review gives everyone present an overview of the current product increment; therefore, it is common to update the product backlog as part of the sprint review.

Activity: Sprint retrospective

The Scrum team improves its own process, always remaining within the Scrum framework. Therefore, at the end of each sprint, the Scrum team meets for the sprint retrospective. The purpose is to review how things went in terms of the process, relationships among people, and the tools. The team identifies what went well and what didn’t go so well, and it identifies potential improvements. The recommended timebox for the sprint retrospective is one hour per week of sprint duration.

Rinse, repeat

This Scrum cycle repeats for every sprint.

To sum up, the Scrum team’s members (the Product Owner, the development team, and the ScrumMaster) collaborate to create a series of product increments during short, time-boxed intervals called sprints. Each increment meets the Product Owner’s acceptance criteria and the team’s shared Definition of Done. The team works from a product backlog. During each sprint, the team begins with sprint planning to produce the sprint backlog, which is a plan for the sprint. The team self-organizes to handle the development, using daily Scrum meetings to coordinate and to ensure that they are producing the best possible product increment. The team performs product backlog refinement to prepare for the next sprint’s planning meeting. It ends the sprint with the sprint review and sprint retrospective, reviewing the product and the process.

Adoption and Future

“Scrum and Agile are similar to Lean Thinking. Scrum is a new way of doing business, but there won’t be that many that reap all of its benefits – such as productivity,  quality, engaged people, and maximized return on investment. American automobile companies were immersed  in running their operations top-down, using the principles of Scientific Management to try to plan everything.
Even while Toyota began to reach their market by using flexible lean techniques, they were unable to adapt. The change in perceptions, culture, and habits was too great.  Many places that are trying to shift from the predictive  approaches of waterfall and command and control are  having the same problems. The change in thinking is  particularly hard. The improved engineering practices  to support the change are also hard. Ken Schwaber predicted that only  those organizations with compelling reason to change  and management with insight and courage will succeed.”
“John Chambers, the CEO of Cisco, put his organization through such a change over the last four years and commented that it was the hardest thing he had ever  done.

“I predict that only 15–25 % of all organizations attempting to use Scrum throughout the enterprise will  succeed in making the change and reaping the benefits. As a result, they will out-compete those who weren’t  able to change and become dominant players in their markets”

In February and March of 2013, Scrum Alliance,  ProjectManagement.com and  ProjectsAtWork surveyed their  readership on use, knowledge  and views of Scrum.  The extensive survey of nearly 500 participants included  in-depth interviews and a literature review.  Participants from more than 70 countries responded to the  survey, with the United States leading at 35% and India second at 12%. The United Kingdom,Canada and Australia accounted  for 4% each, with Germany at  3% and South Korea, Mexico,Belgium, Brazil, Malaysia and New Zealand comprising 2%.

The majority of the participants are from the IT industry (41%),but a good representation camefrom Finance (12%), Government(6%), Healthcare (6%) and Telecommunications (5%), indicating the growing interest in Scrum outside IT. 53% of the participants were from organizations with 1,000 or fewer employees, yet 44% were with companies with annual revenues of $50M to $1B+. 28% were from organizations with ISO 9001 certification, but nearly half (48%) did not have any.

Key Findings
• While 41% of organizations are jumping into Agile waters without requiring a specific certification of their employees, 54% either agree or strongly agree that a certification such as the Certified Scrum Professional (CSP) improves their chances of sustained success.
• Culture is king in the Agile world—and according to a majority of respondents, organizations must create cultures that encourage collaboration in order to deliver value to their customers. This includes fostering self-organized teams and active support from management. Scrum facilitates all of these success factors.
• Scrum is the overwhelmingly preferred Agile method, used by 40% of respondents.The second-most popular method is Kanban, often using many elements of Scrum,a noteworthy trend as organizations seek to find for themselves what works for their specific domains and needs.
How Often and Why They Use Scrum
• In terms of current Agile approaches, Scrum leads the way: 40% of those sampled claimed to be adherents. It was followed by Kanban (15%) and Lean (11%). 19% of the participants used Scrum for up to a quarter of their projects that fall outside of IT, the
majority (38%) of which are in R&D, operations or production.
• 60% of the survey participants used Scrum regularly. 39% used Scrum more broadly throughout their business as one of their project management practices, while 16% used it exclusively for software development projects. 46% of the participants are deploying and managing Scrum projects within a Project Management Office (PMO), and 24% feel that managing and deploying Scrum projects this way is effective and successful.
• In terms of business priorities for Scrum projects, 41% feel that fulfilling customer needs is highest, while meeting budget, time and scope constraints as well as engaging in projects that drive innovation and market share followed with 36%. In terms of providing customer satisfaction while using Scrum, 27% feel having active senior management support is vital,  and 22% say a clear set of business goals for what gets achieved is necessary. That is why over half felt that cultural influences such as an open and collaborative environment—and empowered/self-organizing Scrum teams—are vital for facilitating Scrum; that can only be achieved through active support of management and clear business vision.

Scrum Adoption Factors
• The common belief that the vast majority of organizations adopting and integrating Scrum come from a Waterfall-like background was supported by the survey. 24% use Scrum for some projects while using Waterfall for the rest. 13% use Scrum exclusively, while 31% indicate no use of Waterfall. 23% felt it was a difficult transition from a Waterfall-based method to Scrum. Another perception, stated by 25% of respondents, was that there were no clearly identified metrics to identify and measure the successful delivery of Scrum projects.

Scrum will continue to grow because it is poised for the “Age  of the Customer.”What lies at the  heart of Agile is the  obsessive focus on  service to the  customer—the  mainstay of almost  any business.

Scrum will continue to expand outside of software development. Another adoption of Agile
outside of software development  presents itself in the article “Agile  Reinvents Retail,” published by  John Hitchcock of SandHill.com  on September 2012. The story  describes how a California retail  outlet called Oddyssea used Agile development principles to deploy  its retail operations

Another recent and famous  example of Scrum outside  of software development is  embodied in a sports car called  Wikispeed. Built by Joe Justice, a software engineer by trade, and  a team who entered the 2008  X-Prize competition, Wikispeed  was deployed using Scrum and  crowdsourcing. The team was able  to place tenth in a crowded and  highly competitive environment.
This was done by getting a working  prototype in three months for a  car that can go from 0 to 60 mph  in less than five seconds, weighs  just 1,404 pounds, has a top speed  of 149 mph and gets more than  100 mpg—all using Agile practices  and techniques.

Scrum is well suited for complex problems, where routine cookie cutter solutions does not apply.  In complex domains the ability to probe, explore, sense,inspect, adapt,and respond is critical.

PEARL VIII : Agile Test Automation : Application of agile development principles to Test Automation

PEARL VIII :  Agile Test Automation : Application of agile development principles to Test Automation

Introduction

Today, more and more software development companies are shifting to an agile development approach from traditional waterfall development to keep pace with the current trends.To meet the demands of agile environment, several companies have adopted test automation which resolves some of the issues faced in manual testing and delivers faster results.

Even though test automation provides assistance in many ways, there are several challenges in test automation which can turn out to be a nightmare if it is adopted without appropriate brainstorming and analysis. Here’s a complete list of approaches to make test automation work successfully in agile environment.

The Development methodology agile team follows when creating automated tests is very similar to the process they follow for creating the software being tested. It involves a fair bit of design, coding and testing of its own to get it working correctly. So just like the application itself, automated tests are best developed incrementally – adding new tests and features to the automation framework over several different sprints. It’s important not to aim for the perfect “it-can-do-everything” test framework right at the start, as it would never materialize. Balance the cost vs. ROI well and come up with a bare minimum working solution at the start.

As more companies move from a traditional waterfall software development approach to an agile Methodology, suppliers are offering more test automation tools and services. Automated tests can provide faster feedback than a manual test, reducing rework and long feedback cycles.

Manual testing, particularly manual exploratory testing, is still important. However,  Agile teams typically find that the fast feedback afforded by automated regression is  a key to detecting problems quickly, thus reducing risk and rework.

Research firm Forrester expects testing-as-a-service (TaaS) to emerge as a managed test service from an increase in test automation, as large suppliers such as HP seek to support customers with test automation tools and outsourcing options, including security testing and services for SAP. Better test automation is intrinsic to agile development

Working tests, just like working software, are useful; they build confidence and get everyone excited about the progress being made. Successes, even small ones, make it easier to bring everyone on-board – especially when the test automation solution team have created actually runs and proves to be of real value to the team.

Agile Test team can’t set up automated testing tools without team members who can write code.  Automating tests is software development. It must be done with the same care and thought that goes into writing production code.This poses a significant hurdle for testers who lack coding skills. That challenge is best met by mastering those skills, not by shying away from projects that demand them

Better software is the result of running the right tests and continually re-evaluating which tests are the right ones,

Automated unit tests check the behavior of individual functions/methods and object interactions. They’re to run often, and provide feedback in minutes. Automated acceptance tests usually check the behavior of the system end-to-end. (Although,  sometimes they bypass the GUI, checking the underlying business logic.) They’re  typically run on checked in code on an ongoing basis, providing feedback in an hour  or so. Agile projects favor automated tests because of the rapid feedback they provide.

Agile Testing Quadrant

TheAgileTestingQuadrants.png

In the agile testing quadrant depicted above, The order in which quadrants numbered  has no relationship to when the different types of testing are done. Most projects would start with Q2 tests, because those are where you get the examples that turn into specifications and tests that drive coding, along with prototypes and the like. The quadrants are a taxonomy to help teams plan their testing and make sure they have all the resources they need to accomplish it.The timing of the various types of tests depends on the risks of each project, the customers’ goals for the product, whether the team is working with legacy code or on a greenfield project, and when resources are available to do the testing.

The lower left quadrant represents test-driven development, which is a core agile development practice. The quadrants on the left include tests that support the team as it develops the product.The testing done in Quadrants 1 and 2 are more requirements specification and design aids than what we typically think of as testing.

Unit tests verify functionality of a small subset of the system, such as an object or method. Component tests verify the behavior of a larger part of the system, such as a group of classes that provide some service [Meszaros, 2007]. Both types of tests are usually automated with a member of the xUnit family of test automation tools. We refer to these tests as programmer tests, developerfacing tests, or technology-facing tests. They enable the programmers to measure what Kent Beck has called the internal quality of their code [Beck, 1999].
A major purpose of Quadrant 1 tests is test-driven development (TDD) or test-driven design. The process of writing tests first helps programmers design their code well. These tests let the programmers confidently write code to deliver a story’s features without worrying about making unintended changes to the system. They can verify that their design and architecture decisions are appropriate. Unit and component tests are automated and written in the same programming language as the application. A business expert probably couldn’t understand them by reading them directly, but these tests aren’t intended for customer use. In fact, internal quality isn’t negotiated with the customer; it’s defined by the programmers. Programmer tests are normally part of an automated process that runs with every code check-in, giving the team instant, continual feedback about their internal quality

The tests in Quadrant 2 also support the work of the development team, but at a higher level. These business-facing tests, also called customer-facing tests and customer tests, define external quality and the features that the customers want.

They describe the details of each story. Business-facing tests run at a functional level, each one verifying a business satisfaction condition. They’re written in a way business experts can easily understand using the business domain language. In fact, the business experts use these tests to define the external quality of the product and usually help to write them.

One of the most important purposes of tests in these two quadrants (Q1 and Q2) is to provide information quickly and enable fast troubleshooting. They must be run frequently in order to give the team early feedback in case any behavior changes unexpectedly. All of these tests should be run as part of an automated continuous integration, build, and test process.

Technology-facing tests in Quadrant 4 are intended to critique product characteristics such as performance, robustness, and security. The types of tests that fall into the fourth quadrant are just as critical to agile development as to any type of software development. These tests are technology-facing. Creating and running these tests might require the use of specialized tools and additional expertise.

Quadrant 4 Automation tools

Native database tools

  •  SQL, data import tools
  • Shell scripting 
  • Monitoring tools examples
    • jConsole
      • Application bottlenecks, memory leaks
    • jProfiler
      • Database and bean usage
  • Commercial load test tools
    • Loadrunner
    • Silk Performer
  • Open source test tools
    • jMeter
    • The Grinder
    • jUnitPerf
  • Performance test providers
  • Multiple sites

The short iterations of agile development give your team a chance to learn and experiment with the different testing quadrants

Test Automation Backlog

Maintain a test automation backlog for the project that contains all needed automation tasks and identified improvements. Then target a few items from the backlog every sprint, in no time team will start to see the new regression test suite taking shape. Occasionally, stories from the test automation backlog may require dedicated developer time to implement and consequently some buy-in from the product owner in order to proceed. However, it should not be difficult to convince the product owner about the value of such stories if everyone on team is committed to quality. A test automation backlog could contain a prioritized list of items such as:

  • Parameterize the test environment for test execution.
  • Integrate with Continuous Integration.
  • Enhance reporting mechanism.
  • Provide an option to attach error logs in notification emails.
  • Collect performance metrics for workflow scenarios.
  • Add tests to check for concurrent execution of critical test cases.

Agile test automation pyramid

TA-pyramid 2 Figure 1

In the traditional Test Automation Pyramid view, most if not all of the effort was in developing UI-centric functional tests that explored the application via the GUI. There might be some lower-level tests and a few unit tests, but teams mostly staid at the upper tier.

The agile test automation pyramid is a strategy that attempts to alter the Test automation pyramid and provide the Agile Test automation pyramid depicted in the figure above.

Agile testing relies more on automation. It requires a much greater contribution from developers. And it has a different basic philosophy – to prevent bugs.

The first change is taking a whole-team view. Instead of the testers being responsible for testing AND writing all of the test automation, it becomes a whole-team responsibility. The developers take most of the ownership for unit-level automation, but testers can operate here as well.

At the base of the test automation pyramid is unit testing. Unit testing should be the foundation of a solid test automation strategy and as such represents the largest part of the pyramid.(50 % to 60 % – Unit Tests)

The upper tier focuses on limited GUI-based automation tests. Usually, these are longer running, core customer usage workflows that are best implemented at this level.

Automated user interface testing (0% to 10%) is placed at the top of the test automation pyramid because we want to do as little of it as possible.

The testers typically operate on these, but there are very few tests within this tier. And remember that the developers can operate here as well. The two layers are met by middle-tier automation. This is often the domain of the open source ATDD/BDD tools, such as: FitNesse, Cucumber, JBehave, Robot Framework, and others. The middle tier is the Acceptance Test tier / API Layer tests are performed.

One key thing to note is that traditional automation was often a one-tool operation, with Mercury/HP commercial tooling (Quick Test Professional or QTP) leading that space.  The agile approach is tool agnostic, but also aggregates tools that are appropriate for each layer. Therefor no “one size fits all” thinking is allowed. For example, these are common used tools at each tier:

  1. UI tier:  Selenium, Watir, or traditional QTP
  2. Middle tier:  FitNesse, Robot Framework, and Cucumber
  3. Unit tier:  xUnit family variants for example JUnit or NUnit for Java and .Net respectively

The other consideration is that there are extensions to many of these. For example, both Robot Framework and Cucumber have Selenium plug-ins so that they can ‘drive’ the UI as well as the middle tier. This implies that the middle tier tooling, and automated tests for that matter, can extend or blend into the lower and upper tiers.

Rather than investing in extensive, heavyweight step-by-step manual test scripts in  Word or a test management tool, we capture expectations in a format supported by  automated test frameworks like FIT/Fitnesse. The test could be executed manually,  but more importantly that same test artifact becomes an automated test when the  programmers write a fixture to connect the test to the software under test.

Test-driven development (TDD)

One key benefit of agile software development is the ability to make changes to the code quickly and easily. As the team embraces changing requirements as a part of a living, breathing system, they need to know that their changes will not cause a domino effect of broken code. Ideally, the design of the system will be so simple and well isolated that any possible effects of a change will be readily apparent. In the real world, this is not always the case, especially when working with legacy systems. In this case, something must be implemented to ensure that if a change affects other areas of the system, there will be an immediate red-flag, rather than having to chase down the issues later.

After experimenting with different ways to enable this flexibility, agile teams hit upon the practice of Test Driven Development, which involves producing automated unit tests for production code before you write that production code. Instead of writing tests afterward (or, more typically, never writing those tests), you always begin with a unit test. For every small chunk of functionality in production code, you first build and run a small, focused test that specifies and validates what the code will do. This test might not even compile, at first, because all of the classes and methods it requires may not yet exist. Nevertheless, it functions as an executable specification. You then get it to compile with minimal production code, so that you can run it and watch it fail. You then produce exactly as much code as will enable that test to pass. Sometimes you expect it to fail, and it passes, which is also useful information.

This technique feels odd, at first, to quite a few programmers who try it. It’s a bit like rock climbers inching up a rock wall, placing anchors in the wall as they go. Why go to all this trouble? Surely it slows you down considerably. The answer is that it only makes sense if you end up relying heavily and repeatedly on those anchors (unit tests) later. Those who practice Test Driven Development regularly claim that the benefits of those unit tests more than pay back the effort required to write them.

For Test-First work, you will typically use one of the xUnit family of automated unit test frameworks (JUnit for Java, NUnit for C#, etc). These frameworks make it quite straightforward to create, run, organize, and manage large suites of unit tests. Test Driven Development has grown to be such a useful tool that it is well integrated into most of the major IDEs, including Visual Studio and Eclipse.

Test-driven development (TDD) is an advanced technique of using automated unit tests to drive the design of software and force decoupling of dependencies. The result of using this practice is a comprehensive suite of unit tests that can be run at any time to provide feedback that the software is still working. This technique is heavily emphasized by those using Agile development methodologies

The motto of test-driven development is “Red, Green, Refactor.”

  • Red: Create a test and make it fail.
  • Green: Make the test pass by any means necessary.
  • Refactor: Change the code to remove duplication in your project and to improve the design while ensuring that all tests still pass.

The Red/Green/Refactor cycle is repeated very quickly for each new unit of code.

Follow these steps (slight variations exist among TDD practitioners):

  1. Understand the requirements of the story, work item, or feature that you are working on.
  2. Red: Create a test and make it fail.
    1. Imagine how the new code should be called and write the test as if the code already existed. You will not get IntelliSense because the new method does not yet exist.
    2. Create the new production code stub. Write just enough code so that it compiles.
    3. Run the test. It should fail. This is a calibration measure to ensure that your test is calling the correct code and that the code is not working by accident. This is a meaningful failure, and you expect it to fail.
  3. Green: Make the test pass by any means necessary.
    1. Write the production code to make the test pass. Keep it simple.
    2. Some advocate the hard-coding of the expected return value first to verify that the test correctly detects success. This varies from practitioner to practitioner.
    3. If you’ve written the code so that the test passes as intended, you are finished. You do not have to write more code speculatively. The test is the objective definition of “done.” The phrase “You Ain’t Gonna Need It” (YAGNI) is often used to veto unnecessary work. If new functionality is still needed, then another test is needed. Make this one test pass and continue.
    4. When the test passes, you might want to run all tests up to this point to build confidence that everything else is still working.
  4. Refactor: Change the code to remove duplication in your project and to improve the design while ensuring that all tests still pass.
    1. Remove duplication caused by the addition of the new functionality.
    2. Make design changes to improve the overall solution.
    3. After each refactoring, rerun all the tests to ensure that they all still pass.
  5. Repeat the cycle. Each cycle should be very short, and a typical hour should contain many Red/Green/Refactor cycles.

Characteristics of a Good Unit Test

A good unit test has the following characteristics.

  • Runs fast, runs fast, runs fast. If the tests are slow, they will not be run often.
  • Separates or simulates environmental dependencies such as databases, file systems, networks, queues, and so on. Tests that exercise these will not run fast, and a failure does not give meaningful feedback about what the problem actually is.
  • Is very limited in scope. If the test fails, it’s obvious where to look for the problem. Use few Assert calls so that the offending code is obvious. It’s important to only test one thing in a single test.
  • Runs and passes in isolation. If the tests require special environmental setup or fail unexpectedly, then they are not good unit tests. Change them for simplicity and reliability. Tests should run and pass on any machine. The “works on my box” excuse doesn’t work.
  • Often uses stubs and mock objects. If the code being tested typically calls out to a database or file system, these dependencies must be simulated, or mocked. These dependencies will ordinarily be abstracted away by using interfaces.
  • Clearly reveals its intention. Another developer can look at the test and understand what is expected of the production code.

Rich feature sets provided by the automation framework would essentially reduce the effort put in by the development teams in the TDD workflow. Here are some of the aspects of automation frameworks that can pay rich dividends: Ease of tracking: Unit tests would be stored in a central repository (part of the automation framework) with all the development team members submitting their unit tests to it. Tests would be stored in hierarchical folder structure based on the product features and its components. With this, viewing and tracking of unit tests within and across teams would be smoother. Traceability of unit tests: Automation frameworks can ensure that each product requirement has a unit test associated with it. This ensures that all requirements are developed as part of the TDD process, thus avoiding development slippages.

Improving the development and review process: Automation infrastructure can facilitate tracking of all requirements by associating them with a developer and reviewer(s). This would ensure that development and review processes are organized. Unit test execution: A good automation framework ensures quick running of automated unit tests. The tests could be executed selectively for a component, set of features or the product itself. Reporting of test execution results: Results of the automated unit test for a component/ feature would be sent to the respective developer; this ensures quick reporting and cuts short the response time in refactoring unit tests from the developer. Automation infrastructure components: Automation frameworks could facilitate:

  • Cross-platform testing
  • Compatibility testing

xUnit frameworks

Developers may use computer-assisted testing frameworks, such as xUnit, to create and automatically run the test cases. Xunit frameworks provide assertion-style test validation capabilities and result reporting. These capabilities are critical for automation as they move the burden of execution validation from an independent post-processing activity to one that is included in the test execution. The execution framework provided by these test frameworks allows for the automatic execution of all system test cases or various subsets along with other features.

Fakes, mocks and integration tests

Unit tests are so named because they each test one unit of code. A complex module may have a thousand unit tests and a simple module may have only ten. The tests used for TDD should never cross process boundaries in a program, let alone network connections. Doing so introduces delays that make tests run slowly and discourage developers from running the whole suite. Introducing dependencies on external modules or data also turns unit tests into integration tests. If one module misbehaves in a chain of interrelated modules, it is not so immediately clear where to look for the cause of the failure.

When code under development relies on a database, a web service, or any other external process or service, enforcing a unit-testable separation is also an opportunity and a driving force to design more modular, more testable and more reusable code. Two steps are necessary:

  1. Whenever external access is needed in the final design, an interface should be defined that describes the access available.
  2. The interface should be implemented in two ways, one of which really accesses the external process, and the other of which is a fake or mock. Fake objects need do little more than add a message such as “Person object saved” to a trace log, against which a test assertion can be run to verify correct behaviour. Mock objects differ in that they themselves contain test assertions that can make the test fail, for example, if the person’s name and other data are not as expected.

Fake and mock object methods that return data, ostensibly from a data store or user, can help the test process by always returning the same, realistic data that tests can rely upon. They can also be set into predefined fault modes so that error-handling routines can be developed and reliably tested. In a fault mode, a method may return an invalid, incomplete or null response, or may throw an exception. Fake services other than data stores may also be useful in TDD: A fake encryption service may not, in fact, encrypt the data passed; a fake random number service may always return 1. Fake or mock implementations are examples of dependency injection.

A Test Double is a test-specific capability that substitutes for a system capability, typically a class or function, that the UUT depends on. There are two times at which test doubles can be introduced into a system: link and execution. Link time substitution is when the test double is compiled into the load module, which is executed to validate testing. This approach is typically used when running in an environment other than the target environment that requires doubles for the hardware level code for compilation. The alternative to linker substitution is run-time substitution in which the real functionality is replaced during the execution of a test cases. This substitution is typically done through the reassignment of known function pointers or object replacement.

Test doubles are of a number of different types and varying complexities:

  • Dummy – A dummy is the simplest form of a test double. It facilitates linker time substitution by providing a default return value where required.
  • Stub – A stub adds simplistic logic to a dummy, providing different outputs.
  • Spy – A spy captures and makes available parameter and state information, publishing accessors to test code for private information allowing for more advanced state validation.
  • Mock – A mock is specified by an individual test case to validate test-specific behavior, checking parameter values and call sequencing.
  • Simulator – A simulator is a comprehensive component providing a higher-fidelity approximation of the target capability (the thing being doubled). A simulator typically requires significant additional development effort.

A corollary of such dependency injection is that the actual database or other external-access code is never tested by the TDD process itself. To avoid errors that may arise from this, other tests are needed that instantiate the test-driven code with the “real” implementations of the interfaces discussed above. These are integration tests and are quite separate from the TDD unit tests. There are fewer of them, and they must be run less often than the unit tests. They can nonetheless be implemented using the same testing framework, such as xUnit.

Integration tests that alter any persistent store or database should always be designed carefully with consideration of the initial and final state of the files or database, even if any test fails. This is often achieved using some combination of the following techniques:

  • The TearDown method, which is integral to many test frameworks.
  • try…catch…finally exception handling structures where available.
  • Database transactions where a transaction atomically includes perhaps a write, a read and a matching delete operation.
  • Taking a “snapshot” of the database before running any tests and rolling back to the snapshot after each test run. This may be automated using a framework such as Ant or NAnt or a continuous integration system such as CruiseControl.
  • Initialising the database to a clean state before tests, rather than cleaning up after them. This may be relevant where cleaning up may make it difficult to diagnose test failures by deleting the final state of the database before detailed diagnosis can be performed.

Continuous Integration

CI is a software development practice, where Team members integrate their work frequently, usually with each check-in to the version control system . An automated server builds the system, runs all the unit tests and reports “green” or “red” to indicate if the build is stable/broken.  Building and Testing are done on a dedicated machine to eliminate “it works on my machine” dependencies.

Over the past three years, Ancestry.com, the world’s largest online family history resource, underwent a significant transformation in technology and infrastructure. Starting with the adoption of Agile development practices, the company evolved to a Continuous Delivery model that enables code release whenever the business requires it. Transitioning from large, weekly or bi-weekly software rollouts to smaller, incremental updates has allowed Ancestry.com to increase responsiveness and deliver new features to customers more quickly.

Continuous Integration features

  • Very High ROI. One of the first engineering practices an Agile teams should adopt.
  • Many open source and commercial tools available
  • Keep builds and tests fast (move slow running tests to a nightly build)
  • Adopt a “Stop the Line” mentality (all work stops until a broken build is fixed)
  • Test in a clone of production (Every environmental difference results in risk)

Continuous integration (CI) is the practice, in software engineering, of merging all developer working copies with a shared mainline several times a day. It was first named and proposed as part of extreme programming (XP). Its main aim is to prevent integration problems, referred to as “integration hell” in early descriptions of XP. CI can be seen as an intensification of practices of periodic integration advocated by earlier published methods of incremental and iterative software development, such as the Booch method. CI isn’t universally accepted as an improvement over frequent integration, so it is important to distinguish between the two as there is disagreement about the virtues of each.

CI was originally intended to be used in combination with automated unit tests written through the practices of test-driven development. Initially this was conceived of as running all unit tests and verifying they all passed before committing to the mainline. This helps avoid one developer’s work in progress breaking another developer’s copy. If necessary, partially complete features can be disabled before committing using feature toggles.

Later elaborations of the concept introduced build servers, which automatically run the unit tests periodically or even after every commit and report the results to the developers. The use of build servers (not necessarily running unit tests) had already been practised by some teams outside the XP community. Nowadays, many organisations have adopted CI without adopting all of XP.

In addition to automated unit tests, organisations using CI typically use a build server to implement continuous processes of applying quality control in general — small pieces of effort, applied frequently. In addition to running the unit and integration tests, such processes run additional static and dynamic tests, measure and profile performance, extract and format documentation from the source code and facilitate manual QA processes. This continuous application of quality control aims to improve the quality of software, and to reduce the time taken to deliver it, by replacing the traditional practice of applying quality control after completing all development. This is very similar to the original idea of integrating more frequently to make integration easier, only applied to QA processes.

In the same vein the practice of continuous delivery further extends CI by making sure the software checked in on the mainline is always in a state that can be deployed to users and makes the actual deployment process very rapid.

Twist at AutoTrader

Testing was growing unwieldy for AutoTrader.co.uk, the UK’s #1 automotive website, with multiple tools, lack of  automation, and increasing test maintenance. Enter Twist—the test automation platform that reduced testing effort  by 95%, achieved 89% test coverage, and minimized testing time by 75%. Furthermore, Twist removed the biggest  pain-point, by drastically reducing the overhead of test maintenance. With Twist, the team could add real value and  test new functionality at a rapid pace

Sustaining quality for a dynamic and complex application can end up in a vicious cycle of modifying tests, testing
new functionality, and optimizing existing tests. AutoTrader.co.uk is the UK’s most popular motoring website, and
the fourth busiest search engine, getting over 10 million unique visitors every month. With such high traffic
volumes, optimal test coverage is not just nice-to-have, but a critical necessity. However, testing such a complex,
and ever changing application was a maintenance-heavy, lengthy, and expensive exercise that was further
complicated by difficult testing tools. Auto Trader liked Twist’s support for writing test specifications in plain English,
automating robust regression tests, and integrating with continuous integration (CI) and source control products,
and decided to give it a try. Since 2008, Twist has dramatically reduced AutoTrader.co.uk’s testing effort and
increased test coverage. It has also improved collaboration between business analysts (BAs) and quality analysts
(QAs) on tests, by simplifying technically complex tests to readable English constructs. Twist’s highly refactored,
modular test suite now exhaustively tests the site in 12 minutes, and can easily keep pace with application changes.
Additionally, Twist’s usability makes it highly popular across the teams.

PEARL XII : Kanban : Application of Lean software development to Agile Methodologies

PEARL XII : Kanban : Application of Lean software development to Agile Methodologies.

Most agile m­ethods such as Scrum and XP are already well aligned with lean principles. In 2004, David Anderson pioneered a more direct implementation of Lean Thinking and Theory of Constraints to software development. Under the guidance of experts such as Don Reinertsen, this evolved into what David called a “Kanban system for software development”, and which most people now simply refer to as “Kanban”. ­So while Kanban as applied to software development is new, Kanban as used in Lean Production is over a half century old.

A software development lifecycle process or a project management process could be said to be “lean” if it was observed to be aligned with the values of the Lean Software Development movement and the principles of Lean Software Development.

Bob Charette was invited but unable to attend the 2001 meeting at Snowbird, Utah, where the Manifesto for Agile Software Development was authored. Despite missing this historic meeting, Lean Software Development was considered as one of several Agile approaches to software development. Jim Highsmith dedicated a chapter of his 2002 book to an interview with Bob about the topic. Later, Mary & Tom Poppendieck went on to author a series of 3 books. During the first few years of the 21st Century, Lean principles were used to explain why Agile methods were better. Lean explained that Agile methods contained little “waste” and hence produced a better economic outcome. Lean principles were used as a “permission giver” to adopt Agile methods.

Lean Software Development is an iterative agile methodology originally developed by Mary and Tom Poppendieck. Lean Software Development owes much of its principles and practices to the Lean Enterprise movement, and the practices of companies like Toyota. Lean Software Development focuses the team on delivering Value to the customer, and on the efficiency of the “Value Stream,” the mechanisms that deliver that Value. The main principles of Lean include:

  1. Eliminating Waste
  2. Amplifying Learning
  3. Deciding as Late as Possible
  4. Delivering as Fast as Possible
  5. Empowering the Team
  6. Building Integrity In
  7. Seeing the Whole

Lean eliminates waste through such practices as selecting only the truly valuable features for a system, prioritizing those selected, and delivering them in small batches. It emphasizes the speed and efficiency of development workflow, and relies on rapid and reliable feedback between programmers and customers. Lean uses the idea of work product being “pulled” via customer request. It focuses decision-making authority and ability on individuals and small teams, since research shows this to be faster and more efficient than hierarchical flow of control. Lean also concentrates on the efficiency of the use of team resources, trying to ensure that everyone is productive as much of the time as possible. It concentrates on concurrent work and the fewest possible intra-team workflow dependencies. Lean also strongly recommends that automated unit tests be written at the same time the code is written.

Perhaps the most important Lean activity is to build a value stream map. This means to break down a process into individual steps, and identify which steps add value and which steps do not, thus adding to the waste (muda). The goal, then, is to eliminate the waste and improve the value-added steps (kaizen). An important lean tool helping to manage the work flow is the concept of pull system, which is usually visualized using a Kanban board. In general, we can define the Kanban software process  as a WIP (Work In Process) limited pull system visualized by the Kanban board.

In recent years, Lean Software Development has really emerged as its own discipline related to, but not specifically a subset of the Agile movement. This evolution started with the synthesis of ideas from Lean Product Development and the work of Donald G. Reinertsen and ideas emerging from the non-Agile world of large scale system engineering and the writing of James Sutton and Peter Middleton.  David also synthesized the work of Eli Goldratt and W. Edwards Deming and developed a focus on flow rather than waste reduction . At the behest of Reinertsen around 2005, David introduced the use of kanban systems that limit work-in-progress and “pull” new work only when the system is ready to process it. Alan Shalloway added his thoughts on Lean software development in his 2009 book on the topic. Since 2007, the emergence of Lean as a new force in the progress of the software development profession has been focused on improving flow, managing risk, and improving (management) decision making. Kanban has become a major enabler for Lean initiatives in IT-related work. It appears that a focus on flow, rather than a focus on waste elimination, is proving a better catalyst for continuous improvement within knowledge work activities such as software development.

The Lean Systems Society published its credo at the 2012 Lean Software & Systems Conference. This was based on a set of values published a year earlier. Those values include:

  • Accept the human condition
  • Accept that complexity & uncertainty are natural to knowledge work
  • Work towards a better Economic Outcome
  • While enabling a better Sociological Outcome
  • Seek, embrace & question ideas from a wide range of disciplines
  • A values-based community enhances the speed & depth of positive change

Kanban is an agile methodology for managing the creation of products with an emphasis on continual delivery while not overburdening the development team. Like Scrum, Kanban is a process designed to help teams work together more effectively.

Kanban is a method for managing knowledge work with an emphasis on just-in-time delivery while not overloading the team members. In this approach, the process, from definition of a task to its delivery to the customer, is displayed for participants to see and team members pull work from a queue.

Kanban in the context of software development can mean a visual process management system that tells what to produce, when to produce it, and how much to produce inspired by the Toyota Production System and Lean manufacturing.

The name ‘Kanban’ originates from Japanese and translates roughly as “signboard” or “billboard”. The Kanban method in one of the Kanban methods available today. It was formulated by David J. Anderson is an approach to incremental, evolutionary process and systems change for organizations. It uses a work-in-progress limited pull system as the core mechanism to expose system operation (or process) problems and stimulate collaboration to continuously improve the system. One example of such a pull system is a kanban system, and it is after this popular form of a work-in-progress, limited pull system that the method is named.

In Lean Software Development, the kanban are virtual and often tracked by setting a maximum number for a given step in the workflow of a work item type. In some implementations, electronic systems keep track of the virtual kanban and provide a signal when new work can be started. The signal can be visual or in the form of an alert such as an email.

Virtual kanban systems are often combined with visual controls to provide a visual virtual kanban system representing the workflow of one or several work item types. Such systems are often referred to as “kanban boards” or “electronic kanban systems

Visual control systems are valuable in changing behavior because they display status in an easy-to-see format. This ensures that everyone has a shared understanding of work status and process constraints. Transparency is a key to achieving organizational change

The Kanban method is rooted in four basic principles:

Start with what you do now

The Kanban method does not prescribe a specific set of roles or process steps. The Kanban method starts with the roles and processes you have and stimulates continuous, incremental and evolutionary changes to your system. The Kanban method is a change management method.

Agree to pursue incremental, evolutionary change

The organization (or team) must agree that continuous, incremental and evolutionary change is the way to make system improvements and make them stick. Sweeping changes may seem more effective but have a higher failure rate due to resistance and fear in the organization. The Kanban method encourages continuous small incremental and evolutionary changes to your current system.

Respect the current process, roles, responsibilities and titles

It is likely that the organization currently has some elements that work acceptably and are worth preserving. We must also seek to drive out fear in order to facilitate future change. By agreeing to respect current roles, responsibilities and job titles we eliminate initial fears. This should enable us to gain broader support for our Kanban initiative. Perhaps presenting Kanban against an alternative more sweeping approach that would lead to changes in titles, roles, responsibilities and perhaps the wholesale removal of certain positions will help individuals to realize the benefits.

Leadership at all levels

Acts of leadership at all levels in the organization from individual contributors to senior management should be encouraged.

Kanban Vs Agile

Kanban systems are an approach to scheduling  work. Kanban shares with typical AMs the fact that  requirements are expressed in atomic features (also  known as user stories, work items, Minimum  Marketable Features, or MMF), to be implemented  incrementally. To this purpose, AMs use time-boxed  iterations. – at the beginning of the iteration, the team meets and chooses features from their backlog  that can be done by the end of the iteration. Over the  course of the iteration, they develop those stories,  and at the end of the iteration, they ship them .
Kanban systems are different because they focus on a continuous flow of work, and disregard fixed  iterations. When needed, the team chooses a subset  of features from the backlog and moves them to the  Kanban board. Then, it develops these features one
after the other. Work is delivered as soon as it’s  ready, and the team only works on one – or very few – feature at a time.

Six core practices of Kanban

Anderson identified five core properties that had been observed in each successful implementation of the Kanban method. They were later relabeled as practices and extended with the addition of a sixth.

  1. Visualize
    The workflow of knowledge work is inherently invisible. Visualising the flow of work and making it visible is core to understanding how work proceeds. Without understanding the workflow, making the right changes is harder.
    A common way to visualise the workflow is to use a card wall with cards and columns. The columns on the card wall representing the different states or steps in the workflow.
  2. Limit WIP
    Limiting work-in-process implies that a pull system is implemented on parts or all of the workflow. The pull system will act as one of the main stimuli for continuous, incremental and evolutionary changes to your system.
    The pull system can be implemented as a kanban system, a CONWIP system, a DBR system, or some other variant. The critical elements are that work-in-process at each state in the workflow is limited and that new work is “pulled” into the new information discovery activity when there is available capacity within the local WIP limit.
    WIP limits encourage everyone to work as a team and prevent any one individual from getting too far ahead of anyone else.
  3. Manage flow
    The flow of work through each state in the workflow should be monitored, measured and reported. By actively managing the flow the continuous, incremental and evolutionary changes to the system can be evaluated to have positive or negative effects on the system. Kanban elevates awareness of constraints and forces the team to address them before they can bring additional work into the queue. New work is pulled into the system when there is capacity to handle it, rather than being pushed into the system based on demand. With Kanban, the focus of management and the team becomes the flow of work, not the utilization of the team.
    While Scrum has retrospectives at the end of each iteration to address these items, Kanban explicitly points out constraints in real time and encourages the team to address them as they arise.
  4. Make policies explicit
    Until the mechanism of a process is made explicit, it is often hard or impossible to hold a discussion about improving it. Without an explicit understanding of how things work and how work is actually done, any discussion of problems tends to be emotional, anecdotal and subjective. With an explicit understanding it is possible to move to a more rational, empirical, objective discussion of issues. This is more likely to facilitate consensus around improvement suggestions.
  5. Implement feedback loops
    Collaboration to review flow of work and demand versus capability measures, metrics and indicators coupled with anecdotal narrative explaining notable events is vital to enabling evolutionary change. Organizations that have not implemented the second level of feedback – the operations review – have generally not seen process improvements beyond a localized team level. As a result, they have not realized the full benefits of Kanban observed elsewhere.
  6. Improve collaboratively, evolve experimentally (using models and the scientific method)
    The Kanban method encourages small continuous, incremental and evolutionary changes that stick. When teams have a shared understanding of theories about work, workflow, process, and risk, they are more likely to be able to build a shared comprehension of a problem and suggest improvement actions which can be agreed by consensus.
    The Kanban method suggests that a scientific approach is used to implement continuous, incremental and evolutionary changes.

Common models used are:

  • Theory of constraints (the study of bottlenecks)
  • Deming System of Profound Knowledge (a study of variation and how it affects processes)
  • Lean economic model (based on the concept of elimination of “waste” (or muda, muri and mura)).

Open Kanban

An open source, Agile and Lean based method to deliver value for knowledge work like Information Technology, Software Development, Business, Product Development or Personal organization. On the Lean side it is inspired on the work of Taiichi Ohno (Toyota Production System), Eliyahu Goldratt (Theory of Constraints) and Edward Deming. On the Agile side it takes inspiration from the Agile manifesto signers, and in addition contributions from Alan Shalloway’s Kanban for Teams, Corey Ladas Scrumban and David Anderson’s early Kanban work.

It innovates by making the whole method fully open source and free to improve or modify. Open Kanban was written by Joseph Hurtado, and it has been translated by members of the community to French, Italian, Russian and Ukrainian.

Steps to Get Started

  • Agree on a set of goals for introducing Kanban.
  • Map the value stream.
  • Define some point where you want to control  input. Define what is upstream and who are the upstream stakeholders.
  • Define some exit point beyond which you do  not intend to control.
  • Define a set of work items types based on the work requests that come from the upstream stakeholders.
  • Analyze the demand for each work item type.
  • Meet with the upstream and downstream stakeholders.
  • Create a board/card wall to track the value stream you are controlling.
  • Optionally, create an electronic system to track and report the same.
  • Agree with the team to have a standup meeting in front of the board; invite upstream and downstream stakeholders .
  • Agree to have regular operations review meeting for retrospective analysis of the  process; invite upstream and downstream stakeholders .
  • Train the team on the new board, WIP limits and the pull system. Nothing else in  their world should have changed.

VersionOne Kanban Boards

VersionOne provides a single platform to manage all your projects, from Scrum to Kanban and everything in between, without sacrificing cross-project visibility. Visualize, manage and optimize your workflow using VersionOne’s custom Kanban boards.

With some teams practicing Scrum and others choosing Kanban, it can be challenging for managers to get a clear, integrated view of status across all projects. VersionOne’s Kanban boards make it easy to manage all projects in a single tool without sacrificing cross-project visibility and consolidated reporting. When teams are practicing lean methods within iterations, VersionOne provides the Kanban capabilities to help agile teams visualize and manage their workflow processes.

Agile teams can Create custom Kanban boards using named columns to illustrate each stage of  workflow process, and an expedited track for high priority items. Work items are visually represented using story cards and can be pulled from prioritized queue and moved through the workflow easily using drag-and-drop software functionality.

Managing the Kanban board in VersionOne provides real-time visibility for the entire team with full accessibility for remote members. Agile Teams see at a glance what work items are in progress and where they are in the development process. Visual cues within the Kanban board identify items that are blocked, held up in wait queue, or have exceeded their aging threshold, providing an early indicator of potential problems.

VersionOne software helps maximize  flow of work by setting explicit WIP limits across single or multiple contiguous columns to determine how many items can be worked on at each stage of  process. Visual cues within the Kanban board lets team know when WIP limits are exceeded to trigger conversations and refocus the agile team’s efforts on the right work to ensure a continuous flow.

VersionOne enables to Track and measure the average time to complete a work item using Cycle Cycle Time reporting capabilities. The ability to track cycle time metrics gives visibility into past performance to help  increase the predictability of  delivery and drive continuous process improvement.

The Cycle Time clock starts when work begins on a request and ends when the item is ready for delivery. Cycle time is a mechanical measure of process capability.

VersionOne can be used Identify and resolve bottlenecks using Cumulative Flow reports to see how well items are flowing through the process, Determine where work is stalled and where you should focus improvement efforts.

The Cumulative Flow diagram provides a graphic depiction of how work items are moving through various statuses on the way to being “Done”. It shows the total scope of a project, grouped by status, and thus lets us know how much of that scope is in a particular status at a given time.

As highlightedd, the use of a basic Kanban board (manual and/or virtual for distributed teams)  serves to bring everyone onto the same page, improve focus and prompt questions that lead to revision of policies and practices. When the team’s  experience grows, the Kanban board will change  to expand and refine the workflow representation, thereby reflecting a more sophisticated level