PEARL IX : Refactoring performed to Sustain Application Development Success in Agile Environments

PEARL IX : Refactoring performed to Sustain Application Development Success in Agile Environments

 The term “refactoring” was originally coined by Martin Fowler and Kent Beck which refers to “a change made to the internal structure of software to make it easier to understand and cheaper to modify without altering its actual observable behavior i.e. it is a disciplined way to clean up code that minimizes the chances of introducing bugs and also enables the code to be evolved slowly over time and facilitates taking an iterative and incremental approach to programming and/or design”. Importantly, the underlying objective behind refactoring is to give thoughtful consideration and improve some of the essential non-functional attributes of the software. So, to achieve this, the technique has been broadly classified into following major categories:

1. Code Refactoring (clean-up) : It is intended to remove the unused code, methods, variables etc. which are misleading.
2. Code Standard Refactoring It is done to achieve quality code.

3. Database Refactoring: Just like code refactoring, it is intended to clean (clean-up) or remove the unnecessary and redundant data without changing the architecture.
4. Database schema and  Design Refactoring : This includes enhancing the database schema by leaving the actual fields required by the application.
5. User-Interface Refactoring :  It is intended to change the UI without affecting the underlying functionality.
6. Architecture Refactoring :  It is done to achieve modularization at the application level.

Refactoring is actually a simple technique where you make structural changes to the code in small, independent and safe steps, and test the code after each of these steps just to ensure that you have not changed the behavior – i.e. the code still works the same, but just looks different. Nevertheless, refactoring is intended to fill in some short-cuts, eliminate duplication and dead code, and help ensure the design and logic have been made very clear. Further, it is equally important to understand that, although refactoring is driven by certain good characteristics and shares some common attributes with debugging and/ or optimization, etc., it is actually different because

  •  Refactoring is not all about fixing any bugs.
  •  Again, optimization is not refactoring at all.
  •  Likewise, revisiting and/or tightening up error handling code is not refactoring.
  •  Adding any defensive code is also not considered to be refactoring.
  •  Importantly, tweaking the code to make it more testable is also not refactoring.

Re-factoring Activities – Conceptualized
The refactoring process generally consists of a number of distinct activities which are dealt with in chronological order:

  • Firstly, identify where the software should be refactored, i.e. figure out the code smell areas in the software which might increase the risk of failures or bugs.
  • Next, determine what refactoring should be applied to the identified places based on the list identified.
  • Guarantee that the applied refactoring preserves the behavior of the software. This is the crucial step in which, based on the type of software such as real-time, embedded and safety-critical, measures have to be taken to preserve their behavior prior to subjecting them to refactoring.
  • Apply the appropriate refactoring technique.
  • Assess the effect of the refactoring on the quality characteristics of the software, e.g. complexity, understandability and maintainability, and of the process, e.g. productivity, cost and effort.
  • Ensure the requisite consistency is maintained between the refactored program code and other software artifacts.

Refactoring Steps – Application/System Perspective
The points below clearly summarize the important steps to be adhered to when refactoring an application:
1. Firstly, formulate the unit test cases for the application/ system – the unit test cases should be developed in such a way that they test the application behavior and ensure that this behavior remains intact even after every cycle of refactoring.
2. Identify the approach to the task for refactoring – this includes two essential steps:
– Finding the problem – this is about identifying wheth-er there is any code smell situation with the current piece of code and, if yes, then identifying what the problem is all about.
– Assess/Decompose the problem – after identifying the potential problem assess it against the risks involved.
3. Design a suitable solution – work out what the resultant state will be after subjecting the code to refactoring.
Accordingly, formulate a solution that will be helpful intransitioning the code from the current state to the resultant state.
4. Alter the code – now proceed with refactoring the code without changing the external behavior of the code.
5. Test the refactored code – to ensure that the results and/ or behavior are consistent. If the test fails, then rollback the changes made and repeat the refactoring in different way.
6. Continue the cycle with the aforementioned steps (1) to (5) until the problematic/current code moves to the resultant state.

So, having said about refactoring and its underlying intent, it can be taken up as a practice and can be implemented safely with ease because the majority of today’s modern IDEs (integrated development environments) are inbuilt and equipped with various refactoring tools and patterns which can be used readily to refactor any application/business-logic/middle-tier code seamlessly. However, the situation may not be the same when it comes to refactoring a database, because database refactoring is conceptually more difficult when compared to code refactoring since with code refactoring you only need to maintain the behavioral semantics, whereas with database refactoring you must also maintain information semantics.

Refactoring is the process of clarifying and simplifying the design of existing code, without changing its behavior. Agile teams are maintaining and extending their code a lot from iteration to iteration, and without continuous refactoring, this is hard to do. This is because un-refactored code tends to rot. Rot takes several forms: unhealthy dependencies between classes or packages, bad allocation of class responsibilities, way too many responsibilities per method or class, duplicate code, and many other varieties of confusion and clutter.

Every time we change code without refactoring it, rot worsens and spreads. Code rot frustrates us, costs us time, and unduly shortens the lifespan of useful systems. In an agile context, it can mean the difference between meeting or not meeting an iteration deadline.

Refactoring code ruthlessly prevents rot, keeping the code easy to maintain and extend. This extensibility is the reason to refactor and the measure of its success. But note that it is only “safe” to refactor the code this extensively if we have extensive unit test suites of the kind we get if we work Test-First. Without being able to run those tests after each little step in a refactoring, we run the risk of introducing bugs. If you are doing true Test-Driven Development (TDD), in which the design evolves continuously, then you have no choice about regular refactoring, since that’s how you evolve the design.

Code Hygiene

A popular metaphor for refactoring is cleaning the kitchen as you cook. In any kitchen in which several complex meals are prepared per day for more than a handful of people, you will typically find that cleaning and reorganizing occur continuously. Someone is responsible for keeping the dishes, the pots, the kitchen itself, the food, the refrigerator all clean and organized from moment to moment. Without this, continuous cooking would soon collapse. In your own household, you can see non-trivial effects from postponing even small amounts of dish refactoring: did you ever try to scrape the muck formed by dried Cocoa Crispies out of a bowl? A missed opportunity for 2 seconds worth of rinsing can become 10 minutes of aggressive scraping.

Specific “Refactorings”

Refactorings are the opposite of fiddling endlessly with code; they are precise and finite. Martin Fowler’s definitivebook on the subject describes 72 specific “refactorings” by name (e.g., “Extract Method,” which extracts a block of code from one method, and creates a new method for it). Each refactoring converts a section of code (a block, a method, a class) from one of 22 well-understood “smelly” states to a more optimal state. It takes awhile to learn to recognize refactoring opportunities, and to implement refactorings properly.

Refactoring to Patterns

Refactoring does not only occur at low code levels. In his recent book, Refactoring to Patterns, Joshua Kerievsky skillfully makes the case that refactoring is the technique we should use to introduce Gang of Four design patterns into our code. He argues that patterns are often over-used, and often introduced too early into systems. He follows Fowler’s original format of showing and naming specific “refactorings,” recipes for getting your code from point A to point B. Kerievsky’s refactorings are generally higher level than Fowler’s, and often use Fowler’s refactorings as building blocks. Kerievsky also introduces the concept of refactoring “toward” a pattern, describing how many design patterns have several different implementations, or depths of implementation. Sometimes you need more of a pattern than you do at other times, and this book shows you exactly how to get part of the way there, or all of the way there.

The Flow of Refactoring

In a Test-First context, refactoring has the same flow as any other code change. You have your automated tests. You begin the refactoring by making the smallest discrete change you can that will compile, run, and function. Wherever possible, you make such changes by adding to the existing code, in parallel with it. You run the tests. You then make the next small discrete change, and run the tests again. When the refactoring is in place and the tests all run clean, you go back and remove the old smelly parallel code. Once the tests run clean after that, you are done.

Refactoring Automation in IDEs

Refactoring is much, much easier to do automatically than it is to do by hand. Fortunately, more and more Integrated Development Environments (IDEs) are building in automated refactoring support. For example, one popular IDE for Java is eclipse, which includes more auto-refactorings all the time. Another favorite is IntelliJ IDEA, which has historically included even more refactorings. In the .NET world, there are at least two refactoring tool plugins for Visual Studio 2003, and we are told that future versions of Visual Studio will have built-in refactoring support.

To refactor code in eclipse or IDEA, you select the code you want to refactor, pull down the specific refactoring you need from a menu, and the IDE does the rest of the hard work. You are prompted appropriately by dialog boxes for new names for things that need naming, and for similar input. You can then immediately rerun your tests to make sure that the change didn’t break anything. If anything was broken, you can easily undo the refactoring and investigate.

Example

Add Parameter

A method needs more information from its caller.

Add a parameter for an object that can pass on this information.

Customer                               Customer    
getContact()                                              getContact(data)

inverse of Remove Parameter

Naming: In IDEs this refactoring is usually done as part of “Change Method Signature”

Refactoring a Database – a Major and Typical Variant of Refactoring
“A database refactoring is a process or act of making simple changes to your database schema that improves its design while retaining both its behavioral and informational semantics.
It includes refactoring either structural aspects of the database such as table and view definitions or functional aspects such as stored procedures and triggers etc. Hence, it can be often thought of as the way to normalize your database schema.”
For a better understanding and appreciation of the concept,
let us consider the example of a typical database refactoring technique named Split Column, in which you replace a single table column with two or more other columns. For example, you are working on the PERSON table in your database and figure out that the DATE column is being used for two distinct purposes. a) to store the birth date when the person is a customer and b) to store the hire date when the person is an employee. Now, there is a problem if we have a requirement with the application to retrieve a person who is both customer and employee. So, before we proceed to implement and/or simulate such new requirement, we need to fix the database schema by replacing the DATE column with equivalent BirthDate and HireDate columns. Importantly, to maintain the behavioral semantics of the database schema we need to update all the supporting source code that accessed the DATE column earlier to now work with the newly introduced two columns. Likewise, to maintain the informational semantics we need to write a typical migration script that loops through the table, determines the appropriate type, and then copies the existing date data into the appropriate column.

Classification of Database Refactoring
The database refactoring process is classified into following
major categories:
1. Data quality – the database refactoring process which largely focuses on improving the quality of the data and information that resides within the database. Examples include introducing column constraints and replacing the type code with some boolean values, etc.
2. Structural – as the name implies this database refactoring process is intended to change the database schema.
Examples include renaming a column or splitting a column etc.
3. Referential Integrity – this is a kind of structural refactoring which is intended to refactor the database to ensure referential integrity. Examples include introducing cascading delete.
4. Architectural – this is a kind of structural refactoring which is intended to refactor one type of database item to another type.
5. Performance – this is a kind of structural refactoring which is aimed at improving the performance of the database. Examples include introducing alternate index to fasten the search during data selection.
6. Method – a refactoring technique which is intended to change a method (typically a stored procedure, stored function or trigger, etc.) to improve its quality. Examples include renaming a stored procedure to make it easier to refer and understand.
7. Non-Refactoring Transformations – this type of refactoring technique is intended to change the database schema that, in turn, changes its semantics. Examples include
adding new column to an existing table.
Why isn’t Database Refactoring Easy?
Generally, database refactoring is presumed to be a difficult and/or complicated task when compared to code refactoring. not just because there is the need to give thoughtful consideration to the behavioral and information semantics, but due to a distinct attribute referred to as coupling. The term coupling is understood to be the measure of the degree of the dependencies between two entities/items. So, the more coupling there is between entities/items, the greater the likelihood that a change in one will require a change in another. Hence, it is understood that coupling is the root cause of all the issues when it comes to database refactoring, i.e. the more things that your database is coupled to, the harder it is to refactor. Unfortunately, the majority of relational databases are coupled to a wide variety of things as mentioned below:

■ Application source code
■ Source code that facilitates data loading
■ Code that facilitates data extraction
■ Underlying Persistent layers/frameworks that govern the overall application process flow
■ The respective database schema
■ Data migration scripts, etc.

Refactoring Steps – Database Perspective
Generally, the need to refactor the database schema will be identified by a application developer who is actually trying to implement a new requirement or fix a defect. Then the application developer describes the required change to the concerned DBA of the project and then refactoring begins. Now, as part of this exercise, the DBA will typically work through all or a few of the following steps in chronological order:
1. Most importantly, verify whether database refactoring is required or not – this is the first thing that the DBA does, and it is where they will determine whether database refactoring is needed and/or if it is the right one to perform. Now the next important thing is to assess the overall impact of the refactoring.

2. If it is inevitable, choose the most appropriate database refactoring – this important step is about having several choices for implementing new logic and structures within a database and choosing the right one.

3. Deprecate the original schema – this is not a straightforward step, because you cannot simply make a change retaining the behavior. to the database schema instantly. Instead, adopt an approach that will work with both the old and the new schema in parallel for a while to provide the required time for the other team to both refactor and redeploy their
systems.
4. Modify the schema – this step is intended to make the requisite changes to the schema and ensure that the necessary logs are also updated accordingly, e.g. database change log which is typically the source code for implementing all database schema changes and update log which contains the source code for future changes to the database schema.
5. Migrate the data – this is the crucial step which involves migrating and/or copying the data from old versions of the schema to the new.
6. Modify all related external programs – this step is intended to ensure that all the programs which access the portion of database schema which is for the subject of refactoring must be updated to work with the new version of the database schema.
7. Conduct regression test – once the changes to the application code and database schema have been put in place, then it is good to run the regression test suite just to ensure that everything is right and working correctly.
8. Keep the team informed about the changes made and version control the work – this is an important step because the database is a shared resource and it is minimally shared by the application development team. So, it is the prime responsibility of the DBA to keep the team informed about the changes made to the database. Nevertheless, since database refactoring definitely includes some DDLs, change scripts, data migration scripts, data models related scripts, test data and its generation code, etc., all these scripts have to be put under configuration management by checking them into a version control system for better versioning, control, and consistency.

Once the database schema has been refactored successfully in the application development sandbox (a technical environment where your software, including both your application code and database schema, are developed and unit tested), the team can go ahead with refactoring the requisite Integration, Test/QA, and Production sandboxes as well, to ensure that the changes introduced are available and uniform across all environments.

Refactor Unit Tests

Unit test the current and rewritten code

Unit tests are tests to test small sections of the code. Ideally each test is independent, and stubs and drivers are used to get control over the environment. Since refactoring deals with small sections of code, unit tests provide the correct scope.

Refactor code that has no existing unit tests

When you work with very old code, in general you do not have unit tests. So can you just start refactoring? No, first add unit tests to the existing code. After refactoring, these unit tests should still hold. In this way you improve the maintainability of the code as well as the quality of the code. This is a complex task. First you need to find out what the functionality of thecode is. Then you need to think of test cases that properly cover the functionality. To discover the functionality, you provide several inputs to the code and observe the outputs. Functional equivalence is proven when the code is input/output conformant to the original code.

Refactor to increase the quality of the existing unit tests You also see code which contains badly designed unit tests. For example, the unit test verifies multiple scenarios at once. Usually this is caused by not properly decoupling the code from its dependencies . This is undesirable behaviour because the test must not depend on the state of the environment. A solution is to refactor the code to support substitutable dependencies. This allows the test to use a test stub or mock object. The unit test is split into three unit tests which test the three scenarios separately. The rewritten code has a configurable time provider. The test now uses its own time provider and has complete control over the environment.

Every change in the code needs to be tested. Therefore testing  is required when refactoring. You test the changes at different  levels. Since a small section of code is changed, unit testing  seems the most fitting level. But do not forget the business  value! Regression testing is of vital importance for the business.

Test-driven development (TDD)

Test-driven development (TDD) is an advanced technique of using automated unit tests to drive the design of software and force decoupling of dependencies. The result of using this practice is a comprehensive suite of unit tests that can be run at any time to provide feedback that the software is still working. This technique is heavily emphasized by those using Agile development methodologies

The motto of test-driven development is “Red, Green, Refactor.”

  • Red: Create a test and make it fail.
  • Green: Make the test pass by any means necessary.
  • Refactor: Change the code to remove duplication in your project and to improve the design while ensuring that all tests still pass.

The Red/Green/Refactor cycle is repeated very quickly for each new unit of code.

Key Benefits of Re-factoring
From a system/application standpoint, listed below are summaries of the key benefits that can be achieved seamlessly when implementing the refactoring process in a disciplined fashion:

  • Firstly, it improves the overall software extendability.
  • Reduces and optimizes the code maintenance cost.
  • Facilitates highly standardized and organized code.
  • Ensures that the system architecture is improved by retaining the behavior.
  • Guarantees three essential attributes: readability, understandability, and modularity of the code.
  • Ensures constant improvement in the overall quality of the system.

Justifying the refactoring task might be very difficult, but not impossible. Here are the tips for justifying the need for refactoring.
1. Future business changes will require less time. Refactoring will not give an immediate return but, in the long run, adding features will be less expensive as the code will become easier to maintain. Before refactoring, the code is fit for machine consumption but after refactoring it is fit for human as well as machine consumption.
2. Bugs will be fixed during refactoring. Hidden bugs or logics embedded in complicated unnecessary loops will be exposed, which might result in fixing some longstanding
non-reproducible issues.
3. The current application will have a longer life. Prevention is better than cure. Refactoring can be considered to be a prevention exercise which will help to optimize the structure of the application for future enhancements.
4. There might be performance gains. You cannot promise any apparent or measurable performance gain. But if you are planning to do refactoring to achieve some performance gain, then you should have measurable counters showing the performance of the current app before you start refactoring. And after each change the performance counters should be recalculated to check the optimization.Refactoring may result in a reduction in the lines of code, making it less expensive to maintain in the long run. During refactoring of your algorithm, you should follow the DRY (Don’t Repeat Yourself) principle. Any application
that has survived for 6 months to 1 year will have ample places to remove duplication of code.

Developers do not use the full potential of the refactoring tools available on the market.
This might be due to a lack of knowledge or pressure of timelines. During refactoring, these tools are extremely helpful and valuable as they reduce the chances of intro- ducing an error when making big changes

  • Resharper VIsual Studio Add on for .NET
  • XCode for Objective C #
  • iNTELLIJ idea For Java

Refactoring using the right tools and good software development practices will be a boon for any application’s long life and sustenance. Refactoring is an opportunity to solidify the foundation of an existing application that might have become weaker after adding a lot of changes and enhancements. If you are making changes to the same piece of code for the third time, it means there is some technical debt that you have created and there is a need to refactor this code.

PEARL XV : Beyond Scrum : A Scalable Agile Framework with Continuous Integration

PEARL XV : Beyond Scrum :  A Scalable Agile Framework with Continuous Integration using Test Automation and Build for Large Distributed Agile Projects.

scalable agile process

Scrum is the most popular Agile technique, but it doesn’t scale well. And while Scrum improves the effectiveness of individual teams, productivity gains fall off sharply on large projects with many teams.

Yet Agile methods have been used in very large commercial and open source projects to increase productivity, quality and release frequency. Here are a couple of examples:

  • Facebook incorporates code from 600 developers while delivering two releases per day.
  • The Google Android project utilizes thousands of contributors spread across the world.
  • Flickr reached a level of 10 deployments per day.

Can we learn from these companies and projects? What types of techniques and tools did they use to achieve those results?

This section will cover the problems with Scrum, changes in approach to help scalability and methods for supporting distributed teams and continuous delivery.

Problems With Scrum
Scrum techniques have been very successful in improving the effectiveness of individual development teams, employing concepts like self-directed co-located teams, time-boxed sprints, and regular customer feedback from working software. Yet many organizations have run into obstacles when trying to apply Scrum techniques to large projects. For example, no effective techniques have evolved to coordinate the work of multiple Scrum teams and manage dependencies among them. The “Scrum of Scrums” approach of holding meetings with representatives of every team becomes increasingly time-consuming and unwieldy as the number of teams multiply.

In part, this is because some of the assumptions underlying Scrum are too restrictive for large organizations, or clash with business requirements. Many groups refuse to be limited to co-located teams because they are already distributed, they want to take advantage of the global market for development talent, or simply because many of their employees work from home several days a week.

Many groups need to share key personnel such as architects, UI designers, and database specialists across many projects, and cannot assign them to a single team. Other companies need to fix bugs and release new functionality more frequently than the 2-8 week cycles typical of Scrum teams. This is particularly true of those providing web- and cloud-based applications, where customers expect a constant flow of enhancements.

This is not to say that Scrum practices have no place in a large development environment. But it is now clear that many organizations need to go “Beyond Scrum” and find new practices to manage distributed contributors and large, complex projects.

Changes in Approach That Help Scalability
The large commercial and open source projects that have successfully scaled Agile typically depart from conventional Scrum practices in several areas.

No Scrum meetings: Sprint planning meetings, retrospectives and Scrum-of-Scrum meetings are time-consuming, usually require that everyone be in one room, and usually don’t do a very good job of coordinating across teams. That’s why large-scale projects find ways to use online collaboration and planning tools to coordinate work within and across teams, with fewer meetings and conference calls.

“Pull,” “continuous flow,” and “publish what’s ready”: Although Scrum practices are far more agile than the waterfall methods they replaced, they still impose a degree of inflexibility. Once a sprint plan is complete, the features to be delivered are fixed, and so is the time frame to deliver them (usually 2-8 weeks).

Scalable projects typically use pull and continuous flow techniques (especially Kanban), so developers are always working on the highest priority tasks, and new functionality can be released as soon as it is ready.

Code review workflows (long used by open source projects) can be used to select contributions from hundreds of contributors and publish what’s ready. By helping organizations scale to more contributors and release more frequently, code review workflows can become a key building block of Scalable Agile.

Diverse contributors: Classic Scrum practices are designed for co-located teams of 8-10 members. But, in reality, large projects need to incorporate work from individual contributors, shared resources (e.g., architects and DBAs), outsourcing companies, and business partners as well as teams. Collaboration tools and code review workflows are central to meshing the work of these diverse contributors.

A Scalable Agile Process Framework
The question, then, is how can we apply the new approaches as a coherent whole?

Instead of each Scrum team having its own backlog, product management (or product owners) maintain a single project-wide backlog, with tasks sorted in priority order.

At any time, contributors can pull tasks from the top of the backlog into their own “Current Work” list. Ideally they will pull the highest-priority task, but if they do not have the necessary skills or resources they can move down the stack to find another high-priority assignment.

This process ensures that the highest-priority tasks are addressed first. If an urgent bug fix or a key customer feature request is placed at the top of the backlog it will receive immediate attention, instead of waiting for the next sprint.

Contributors can be individuals, teams, departments, or even entire organizations like an outsourcing firm or a business partner. There is no expectation or requirement that the tasks be done by Scrum teams of 8-10 members. This allows organizations to call on the talents of all kinds of individuals and companies, and in fact conforms to the reality of most large projects today.

Tasks are then managed using Kanban or lean principles. Typically this means that each person is working on one task at a time (i.e., the team has a work-in-process limit of one task for each person on the team).

Kanban principles ensure that once tasks are started they are completed as quickly as possible, which means that they can be released sooner, and also that other tasks which depend on the first task can be started sooner.

When tasks are completed, the contributor pulls in on-demand resources to build and test a release with the new code. This provides immediate feedback to the contributors, and allows them to catch and fix bugs right away. It also makes features available faster, because there is no wait for centralized build and test systems.

Finally, once new code submissions have been tested successfully, they can be pulled through a merge process into a staging area or into a final build. This means that a new version of the software can be assembled and released at any time, with whatever bug fixes and enhancements are available at that moment.

What Does This Accomplish?
How exactly does this Scalable Agile process framework address the shortcomings of Scrum and provide more scalable, responsive development efforts? Here are a few of the advantages:

  • There can be many types of contributors, including (but not limited to) conventional Scrum teams.
  • There is no need to spend time estimating tasks precisely, doing detailed sprint planning, or having long meetings to coordinate assignments across teams. As long as the backlog is maintained in priority order the highest-priority tasks will be addressed first.
  • Once tasks are started they are completed in the least possible time, meaning they can be released faster and dependent tasks can be started sooner.
  • Software quality is better, because test feedback is available as soon as a task is complete. Bugs can be fixed when it is clear what changes caused the problem, and when the code is fresh in the mind of the developer. Also, quality assurance does not become a bottleneck, a situation which often leads organizations to cut corners on testing (leading to yet more quality problems).
  • New versions of the application can be assembled and released at any time according to business demand. With sufficient automation this can be daily or even several times a day.

Scrum is the most popular Agile techniques, but it doesn’t scale well. And while Scrum improves the effectiveness of individual teams, productivity gains fall off sharply on large projects with many teams.

In the earlier section, methods on how to apply agile techniques to distributed teams and large projects was dealt. In the following section,  Tools and techniques for managing distributed teams will be addressed.

Some of the processes and tools needed to manage the Scalable Agile process framework are addressed in the following paragraphs.

The first of these “building blocks” is support for distributed teams. Large development organizations are almost always distributed because they have (1) business units and business partners in multiple locations, (2) “outsourcing” groups in different countries, (3) decided to take advantage of the global market for development talent, (4) remote employees who work from home.

So how can organizations support distributed teams well enough to reduce the need for face-to-face meetings?

Online Agile Planning
Online tools can replace paper-and-pencil planning exercises and physical whiteboards. This allows team members worldwide to create and maintain an overall project backlog, pull tasks to individual teams and contributors, move tasks through the steps in a Kanban process, and view tasks ready to be pulled into a release.

The  an online Agile planning tool is used for managing a central backlog and pulling tasks into “current” task buckets for individual teams, and an online card wall that can replace the physical variety.

online planning tools
Online planning tools can replace paper plans and physical white boards.

Online Collaboration

Development team members can collaborate most easily when they are in the same room, but online tools can provide a close approximation. Such tools include online standup reports, wikis, chat and IM products, and video and teleconferencing systems.

Another type of online tool, an activity stream gives developers real-time visibility into the activities of other team members—activities like code commits, new tickets, comments added to tickets, code reviews and posts on wikis.

activity stream

An activity stream shows commits, comments, and other events.

Global Code Management

Global collaboration can be undermined if developers need to share large repositories and large files over long distances and performance is slow.

Some of the technologies that Tool vendor Perforce uses, like proxies and replication, ensure that files are available immediately in remote locations . These solutions ensure that data is available where needed without artificial boundaries that impede sharing and collaboration.

Perforce technologies ensure that distributed team members don’t have to wait to get large repositories and files.

Decentralized Code Management
Developers often want highly decentralized code management so they can create their own local test branches and work independent of centralized corporate resources.

Development managers, however, want to maintain control over and visibility into activities at remote locations.

Git Fusion from Perforce answers both needs. Developers can quickly clone their own repositories and work in private Git repositories on their local systems, with easy code sharing between teams and products.

select directories

Release managers can make selected directories visible to Git users.

Release managers can model an entire product development effort with Perforce streams and branches, apply access controls, and control how much history and which files are cloned into new Git repositories . As changes are accepted, the enterprise release model guides changes to the right places: older releases, customizations, and parallel development efforts.

When developers commit code to the Perforce repository, the Perforce shared versioning service makes the changes visible to everyone and maintains a strong system of record about the source and nature of all changes.

The earlier paragraphs described the challenges of scaling large and distributed Agile teams, and investigated the tools and strategies that resolve them. Once the problem of scaling Agile development has been addressed, however, pressure is then applied next to the people and processes that are tasked with delivering or deploying the resulting product. Agile workflow is only successful once efficiency is attained in all stages of the workflow. This paragraph will cover the second building block of Scalable Agile: Continuous Delivery.

ScrumBan

ScrumBan is a relatively easy first step for Scrum teams that want to move in the direction of Continuous Delivery.

Teams using ScrumBan work within a time-boxed sprint. But unlike conventional Scrum practices, a work-in-process limit is adopted, so team members are focused on finishing one task at a time. At a certain point in the sprint a “triage” process identifies which tasks can be completed within the time box, and drops the others from the sprint plan. At that point there is a “feature freeze,” and the remainder of the sprint is devoted to completing the tasks specified by the triage process.

scrumban

ScrumBan represents a step toward Continuous Delivery because it emphasizes completing a small number of tasks as quickly as possible. Development teams avoid the pitfalls involved in pulling out all the stops to deliver the entire sprint plan, regardless of the cost in terms of quality and delays.

On-Demand Merge and Test by Contributor
The conventional software release process creates a huge bottleneck at the test phase. All teams send their contributions to a central QA team, which creates and tests a “release candidate.”

In traditional release processes, the QA lab becomes a serious bottleneck.

In theory this workflow makes very efficient use of the QA team and test systems, however:

  • It takes a long time to run all of the tests.
  • It is hard to debug and troubleshoot many code changes at once, especially if they may be interacting with each other.
  • Errors uncovered during the integration phase may require costly rework by several contributors.
  • The test lab becomes a huge bottleneck near the end of each sprint, causing stress and leading to sloppy testing practices.
  • Releases are delayed until the entire release candidate has been completely tested and debugged.

But what if each team can build and test based on just its own contributions?

There can be a different approach to testing. In this scenario each team and contributor has access to test resources. QA team members act as advisors and facilitators rather than being charged with managing all of the testing themselves. When a development team finishes a set of changes, the team then pulls a copy of the production version onto the test system. Changes can then be merged into this private production version, built and tested locally.
Each team pulls a product version, merges its changes, and performs its own tests.

team pulls

If testing uncovers problems, these can be solved right away by development, without worrying about interactions with changes from other teams. Multiple teams and contributors can now test and debug contributions independently, and submit them to the central staging area when ready
Teams submit tested contributions when they are ready; releases can be assembled at any time.

The advantages of this approach include:

  • QA is no longer a bottleneck—teams test independently, when they are ready.
  • It is easier to debug and troubleshoot problems, because each group is observing only its own changes to a previously tested production version.
  • Releases can be assembled at any time, constructed from whatever contributions have been tested and submitted.

These capabilities are not easy or cheap to implement. They require a considerable investment in automated build and test environments, which for distributed teams must be provided in the cloud. They also require that code merge and management be a fast and easy part of any developer’s daily work. A simple and easy-to-automate merge framework like Perforce Streams  provides merge notifications, merge pathway guidance, and intuitive tools.

In a complicated project consisting of several components, Perforce’s visibility over any part of the project also helps development teams share and reuse code. These teams can quickly adapt to a changing project structure, even if they are working in distributed repositories via Git Fusion. A merge in this environment would never require a complicated action that spans several independent repositories.

These capabilities are an indispensable aspect of Scalable Agile, because they allow very large numbers of teams to contribute to a project without overwhelming build and test resources.

Code Review Workflow
Another major challenge for Continuous Delivery is how to merge a growing number of contributions into production releases. How can you organize the flow of contributions from many sources? How can you decide when to assemble the next release? How do you avoid creating a bottleneck at the point where the contributions come together?

One very useful method is a code review workflow similar to those used in open source projects. In these projects hundreds of contributors might submit code and thousands might test it. Typically a core group of “maintainers” reviews submissions and selects the ones that will be included in the next release.

A code review workflow can be utilized in commercial environments as well. for example, the Assembla ticketing tool includes a merge request feature that allows contributors to submit code changes for review. Designated reviewers can review the submissions, hold online discussions about them, vote for or against accepting them, and make immediate decisions to accept or reject them

This code review workflow lets organizations manage the code review process and delegate the decision making for accepting contributions and assembling releases, which prevents these activities from becoming bottlenecks.

A code review workflow allows designated reviewers to vote on which contributions to include in the next release.

Streams for Managing Multiple Versions
Another common challenge among large projects is maintaining multiple releases and managing custom versions for individual customers.

Software vendors, for example, usually need to support several releases of an application at once. Bug fixes might need to be applied to many (but not all) of the supported releases. Enhancements to the current release might be retrofitted to the previous release and added to the upcoming release under development. Similarly, a service provider or enterprise IT department might be maintaining customized versions of an application for different customers or different business units within the enterprise.

perforce streams

It is much easier to navigate these complex scenarios with a tool like Perforce Streams. Perforce Streams not only helps development managers visualize the relationships between releases and versions, it guides release managers on where and when to apply bug fixes and feature enhancements when they are ready to be merged

Perforce Streams provide adaptable workflow for teams and promote efficiencies such as code re-use, automated merging, fast context switching, efficient workspace updates, and inherited workspace and branch views. An innovative addition to the Perforce branching and merging toolset, streams eliminate overhead, simplify common processes, and increase agility and scalability. In projects with a large volume of data, the time and performance savings are considerable.

Perforce Streams helps deploy bug fixes and enhancement across multiple releases and custom versions.

The typical perception of Agile development methodologies is that their benefits and promise are reserved for small, co-located teams. However, in the above sections we have seen how many, if not all, of the traditional Agile practices can be improved to the benefit of not only large teams, but large distributed teams as well. Ironically, this scalability has been achieved by employing the very processes and tools that the Agile Manifesto preaches against. However, while the tools enable scalability, they never require the sacrifice of developers’ freedoms or their ability to interact.

In these final paragraphs , tools will again be the focus for providing the solution for scaling one of the essential requirements of any Agile workflow: Continuous Integration with build and test automation. All the ideas addressed in the above paragraphs  will be reviewed along with detail on  how they fit together to make Agile scalable.

All of the examples provided can be implemented using software from Perforce, Assembla, Git and Jenkins.

Methods for Providing On-Demand Infrastructure

In a large project, the trickiest and costliest problems are found only when individuals put together all the pieces. It pays to find and fix these integration problems as early and often as possible.
Continuous Integration is a set of best practices in software development that supports project integration on a rapid, repeated basis . They are:

  •  Maintain a code repository
  •  Automate the build
  •  Make the build self-testing
  •  Everyone commits to the mainline every day
  •  Every commit (to the mainline) should be built
  •  Keep the build fast
  •  Test in a clone of the production environment
  •  Make it easy to get the latest deliverables
  •  Everyone can see the results of the latest build
  •  Automate deployment

The goal of is to perform a project-wide integration as often as possible. Striving to achieve this shapes your infrastructure, development practices, and attitudes.
Attitude is important. The entire team must commit to achieving successful integration at all stages of the project. For instance, a development task is not considered to be “done” until the feature appears in the integration build and is proven to work there. That shared commitment should make developers uneasy when they risk divergence by working in isolation for any lengthy period, e.g. when they are using an old build as a stable development base, or when they commit their changes their only infrequently. We cannot emphasize enough the importance of frequent integration. It really does reduce project risk.

Continuous Integration Tools in the Cloud
Organizations clearly need to invest in automated build and test processes if they want to scale up and deliver features faster and release more frequently. This investment can be expensive, but manual methods are obviously not scalable. Also, automated build and test processes tend to produce much higher software quality.

And if teams and contributors are highly distributed? Then the build and test tools must be accessible online, in the cloud.

Automated test tools like Jenkins can be integrated into the code review and merge workflows described earlier paragraphs. Whenever a contribution is accepted, a new version can be built and a series of automated tests can be run against it. Tools like Jenkins will then provide developers and the QA staff with detailed information on test results. Results from the test tool can even be used to vote to accept or reject contributions, as part of the code review workflow.

Automated test tools can provide detailed information on test results, and even vote to accept or reject contributions.

jenkins ci

Managing the impact of on-demand continuous integration is a logistical challenge for the version management service. Perforce addresses this challenge by providing flexible configurations of proxies and replicas to meet a variety of build demands.

supporting ci

In the earlier paragraphs,  some of the shortcomings of Scrum, such as lack of techniques to coordinate teams, assumptions about co-located teams and fixed release cycles that are unrealistic for many organizations, and a tendency to spend too much time in planning and coordination meetings was covered.

Continuous, iterative development is supported for different workflow methodology employed. With Perforce,  can:

  • Build and confirm your work from a private workspace before submitting your code
  • Execute automated builds and tests on specific branches upon check-in
  • Improve the quality of software and delivery time to market
  • Popular continuous integration tools, like Electric Commander from Electric Cloud, Parabuild from Viewtier, and Anthill Pro from UrbanCode, all support their own integrations with Perforce.

 A Scalable Agile process framework featuring the following was outlined:

  • A single prioritized backlog for all teams, so high-priority tasks always receive immediate attention.
  • Kanban processes with WIP limits, so teams don’t have to spend a lot of time in release planning, and to ensure that individual tasks are completed as quickly as possible.
  • On-demand resources, so each team can build and test its own contributions quickly and avoid making the QA lab a bottleneck.
  • A code review process that allows designated reviewers to accept or reject a large number of code submissions.
  • A “take what’s ready” approach to releases, so organizations could provide new functionality as frequently as required by customer needs and expectations.

The processes and tools that could facilitate Scalable Agile was discussed. These include online Agile planning tools, online collaboration, global code management, decentralized code management, ScrumBan processes, tools for on-demand merging and testing by contributors, code review workflows, stream-based tools for managing multiple releases and custom versions, and continuous integration tools provided in the cloud.

While the journey to Scalable Agile may be a long one, each of the steps down the path provides immediate benefits. The growing development groups consider:

  • Implementing ScrumBan, to start moving toward lean methods.
  • Deploying online planning and collaboration tools, to improve the effectiveness of distributed teams and contributors.
  • Deploying advanced code management platforms, to support distributed development and manage multiple releases and versions.
  • Begin investing in Continuous Integration and on-demand build and test systems.
  • Adjust the dial on continuous delivery gradually, to allow time for all your teams to adjust.

PEARL XVI : SCRUMBAN – A tool for software production and Support

PEARL XVI : Scrumban is a combination between Scrum and Kanban methodologies for increased applicability and versatility tool for software production and support focused companies.

Scrum is an iterative and prescriptive process for building software using the Agile methodology. A development team plans and commits to completing a certain amount of work in a certain time period called a sprint. At the end of the sprint, the team reviews the work with a product owner. They then hold a retrospective to analyze their processes during the sprint, and determine what can be improved next time.

Scrum can be an effective tool for introducing teams to Agile. Its more rigid structure provides a framework of understanding that is far easier to grasp than the loose nature of Kanban might be for teams used to rigidly planned waterfall-style projects.

The daily standup, planning, review, and retrospective meetings are excellent touch points for periodic checking in with the work that is happening and reviewing with stakeholders. The retrospective itself is the crown jewel of the Scrum framework, and is what enables teams to focus on continuous improvement, or Kaizen.

One of the primary motivations for moving a team to Scrum is to get away from the often restrictive and inefficient processes of the waterfall model. In this model, there is frequently too much time spent on planning and designing before any work begins. After that, teams find themselves unable to respond to change later in the project as the developing and testing proceeds. All of this is why waterfall is a sub-optimal process for software development.

But look at the makeup of a typical Scrum sprint:

  1. Team plans the work that will be worked on over the next sprint.
  2. During planning, teams try to design as many features as possible, so that they can more accurately estimate what they can complete during the Sprint.
  3. During the Sprint, the team develops and then tests their user stories.
  4. At the end of the Sprint, the Product Owner reviews the work completed, and decides which of the stories are shippable and ready for production.

What does this mean? Scrum is a series of miniature waterfall projects wrapped up into iterations called “Sprints.” This is a push-based system, where another solution would be to use a pull-based one. Enter Scrumban.

Scrumban can give teams the power to adapt and change to stakeholder and production needs, without feeling overburdened by their project methodology. It is designed to remove metrics that encourage undesired outcomes. It can restore working time to the team, and avoids unnecessary meetings. And most importantly, it can limit the team’s work in progress so that they can finish what they start to a high standard. Scrumban can remove overhead stress for the development team, increase efficiency, and increase the overall satisfaction for the customer.

Scrumban combines the best features of both methods. It compounds the prescriptive nature of Scrum and the process improvement of Kanban, allowing teams to become Agile and to continually improve their process. Scrumban is becoming especially popular in services industries, where project development and maintenance goes together.

Scrumban is a process for evolving a Scrum instance to be more Lean oriented, supplemented with core Kanban practices such as: visualization, work-in-process limits, work flow management, and making policies explicit. The basic Scrumban implementation principles include:

  • Start with all the ceremonies, artifacts and roles you use now.
  • Agree to pursue improvement towards a more effective process.
  • Respect current roles, responsibilities and job titles.

Scrumban as a combination of a Kanban and Scrum method

Scrum-ban is a software production model based on Scrum and Kanban. Scrum-ban is especially suited for maintenance projects or (system) projects with frequent and unexpected work items or programming errors. In such cases the time-limited sprints of the Scrum model are of no appreciable use, but Scrum’s daily meetings and other practices can be applied, depending on the team and the situation at hand. Visualization of the work stages and limitations for simultaneous unfinished work and defects are familiar from the Kanban model. Using these methods, the team’s workflow is directed in a way that allows for minimum completion time for each work item or programming error, and on the other hand ensures each team member is constantly employed.

To illustrate each stage of work, teams working in the same space often use post-it notes or a large whiteboard. In the case of decentralized teams, stage-illustration software such as Assembla, ScrumWorks, TargetProcess, Rational Team Concert, Eylean board, OnTime, Kanban Tool or JIRA in combination with Jira Agile can be used to visualize each team’s product backlog items, defects and tasks divided into separate phases.

In their simplest, the tasks are categorized into the work stages:

  • Unstarted
  • Ongoing
  • Completed

If desired, though, the teams can add more stages of work (such as “defined”, “designed”, “tested” or “delivered”). These additional phases can be of assistance if a certain part of the work becomes a bottleneck and the limiting values of the unfinished work cannot be raised. A more specific task division also makes it possible for employees to specialize in a certain phase of work.

There are no set limiting values for unfinished work. Instead, each team has to define them individually by trial and error; a value too small results in workers standing idle for lack of work, whereas values too high tend to accumulate large amounts of unfinished work, which in turn hinders completion times. A rule of thumb worth bearing in mind is that no team member should have more than two simultaneous selected tasks, and that on the other hand not all team members should have two tasks simultaneously.

The major differences between Scrum and Kanban are derived from the fact that, in Scrum, work is divided into sprints that last a certain amount of time, whereas in Kanban the workflow is continuous. This is visible in work stage tables, which in Scrum are emptied after each sprint. In Kanban all tasks are marked on the same table. Scrum focuses on teams with multifaceted know-how, whereas Kanban makes specialized, functional teams possible.

In Kanban, Workitems deemed  high priority and needed as soon as  possible have a way of being addressed  ahead of current work. Periodic reviews  of process, tools, team performance, and
WIP limits to find improvements and  eliminate inefficiencies or waste.

When to use Scrumban?

Are you considering implementing Scrumban in your team or business? You should take it into account if you are doing one of these:

  • Projects maintenance,
  • Event-driven work (support, hardening),
  • Problematic projects management (projects with unexpected user stories and bugs),
  • New product development (work preceding sprint development or following sprint development),
  • Continuous management improvement.

Scrumban board gives an easy view of process workflow. It informs about the number of items the team is currently working on as well as already finished ones. It influences the improvement of accountability, responsibility, communication and performance results.

 The example of a Scrumban board

Scrumban is a pull-based system, where the team no longer plans out the work that is committed to during the planning meeting, and instead continually grooms the backlog. The same Scrum meetings (planning, review, and retrospective) can and should still take place, but the cadence of them can be more context-driven. The real key to moving to Scrumban, though, is ensuring that work in progress (WIP) is still limited.

Work-in-progress limits, not Sprints. With Scrum, the amount of work that is ongoing is limited by the Sprint time commitment. But in Scrumban, with no specific time commitment, the team must limit itself through the use of WIP limits on columns within their task board. The goal is always to move tickets in a flow from left to right on the board. If too many issues are in progress, the team is at risk of not finishing anything to high quality standards. Instead, there should be a maximum number of tickets allowed per column. If the number of tickets in that column ever exceeds the maximum, the entire team should swarm onto that column and help move tickets on. This should happen no matter what functional role a team member fills.

Key Scrumban Terms
• Work Backlog – Equivalent to Scrum’s Product Backlog with minor differences for how  Workitems are tracked.
• Workitem– The individual descriptions of functionality needed by the business that either define enhancements or defects.
• Workline – The series of stages in the software development process that pull work from the  Work Backlog and produce shippable business value.
• Stages – The common and discrete steps required to deliver software within an organization.

The Weekly Timeblock (WTB)

The Weekly Timeblock is a reoccurring meeting used for multiple purposes. It should be set up in the  middle of the week (Wednesday are best) so as to avoid holidays and vacation days that typically  occur at the beginning and end of each workweek.
Two to four hours is recommended and set to occur in the afternoon so any preparation work can be  completed ahead of time. The team will use the time they need and leave the rest. Having a projector,  screen, and ample whiteboard space will be conducive for any technical discussions and problem  solving.
The first use of the Weekly Timeblock is for demonstrating any completed work. This is the time  where Developers can show the Product Owner what was completed and get Acceptance. Completed  work can be demonstrated outside of the meeting anytime but it is good for the team to have  every one witness the conclusion of work.
Once Workitems have been demonstrated, the team should reflect on how their development process  is going, any tools they use, and review the data on how Workitems are flowing through the  Workline. They should have frank discussions about any new innovation that should be reinforced or  impediments to be overcome. Clear action items and their owners should be identified along with a  target resolution date.
After the review of recent activity, the team should have a preview of upcoming Workitems the  Product Owner is preparing. The Developers should seek to understand the intent of each item,  asking questions and getting clarity so everyone is familiar with what’s needed. The Developers  should also provide feedback on Workitem scope, structure, and the optimal execution order so the  Product Owner can make any needed adjustments to content and priority.

Standup meetings. The Daily Standup meeting is a concept borrowed from Scrum and serves the same purpose; to coordinate activities and identify impediments. A daily reoccurring meeting should be set up for 30  minutes where the first half is focused on the team members updating everyone on what they did  since the last meeting, what they intend to do before the next meeting, and any impediments that are  blocking, or threaten to block progress on their work. The second half is for any impromptu  discussions necessary to address questions or plan activities. The meeting should be held near the Board so discussions focus around work in progress and updates can be made publically. Ideally the  meeting should occur early in the day, just after everyone has had a chance to get to work, reorient themselves, and think about what they’ll focus on.

In practice, this boils down to redundant statuses that recount information available on the team’s task board. For Scrumban, a more effective method is to refocus on the flow of tickets on the board. That same pattern of yesterday/today/blocked can be transferred to the tickets themselves—the group moves through each column and briefly discusses each ticket and what is necessary to move that ticket rightward on the board. This provides far more context to the team and informs everyone of any major architectural or design decisions.

Metrics. Metrics can certainly be useful, but they are often abused by managers and business stakeholders who want to unnaturally simplify a complex process into a one-dimensional number. Velocity, the amount of story points a Scrum team completes in a single Sprint, is such a metric that incentivizes lower quality at the end of a Sprint as a team scrambles to finish every last story they committed to. When the number fluctuates, as is common with a newer team, the stakeholders begin to question the outputs of the team, and even the effectiveness of Agile itself.

Instead of velocity, a useful Scrumban metric is cycle time. This is the length of time a ticket takes to complete, measured from when it is first began. Over time, a statistical analysis of all tickets in the project can yield a mean cycle time and standard deviation. This can be a useful planning tool at a macro level, as it is trivial to add up the number of stories and multiply by mean cycle time.

The Work Backlog under Scrumban is basically the same as with Scrum. It is a collection of User  Stories and Acceptance Criteria written and maintained by the Product Owner. These individual Workitems describe the business value desired.

To keep the collection organized, additional information is associated with each item:
• ID–A unique identifier (e.g. B#042) for easy searches and reference.
• Project –A label to group a set of work items together according to a theme or purpose.
• Title –A concise, action-oriented description of the business value desired.
• User Story –A formulaic description of who will benefit from the item, what exactly is  wanted, and why it is wanted. The User Story provides the context behind the request and  enables the Developers to understand the goal, and not just take orders.
• Acceptance Criteria –A list of the specific conditions required for the Workitem to be  considered complete.
• Status –An indicator of the readiness of the Workitem, when it is being worked on, and the  final disposition.

When the progress of a Work item through the Workline is blocked

As a team is working on items, it will be fairly common that progress from one stage to another is blocked. These blockages can be internalto the team where the work was more challenging than expected. Whatever the cause the team should “swarm” to help wherever possible to remove the blockage. All team members can assist to one degree or another, even crossing the traditional role boundaries of Coders & Testers to help test, code, prepare environments, research or write documentation.
When a blocking issue is external to the team, such as when a required server hasn’t yet been delivered or subject matter experts are needed for questions, the Scrum Master should attempt to resolve it. The rest of the team can be engaged as needed to help with external issues.
If a Workitem is blocked completely with no resolution in sight, it should be pulled from the Workline and its status changed from ‘In Progress’ to ‘New’ or ‘Draft’ depending on what is needed before the team can pull it back into the Workline again.

Workitem needs to be completed as soon as possible
There are two ways to address this need. First, the Product Owner should be continually maintaining  the order of the Work Backlog such that the items the business considers the most valuable are at or  near the top. It is these top items that the team reviews and makes ‘Ready’ through discussions and  feedback and are pulled into the Workline as capacity opens up.
If there a more urgent need for a work item to be completed than what the standard process  provides, the item’s priority can be changed to expedited and be fast-tracked through to completion.

Defects in Scrumban

Defects are a natural part of any software development effort and dealing with them is straightforward. When defects associated with current Workitems are found they should be addressed before the item is considered complete and ready for demonstration.
Assuming the defect was found during the Testing stage and it is relatively minor, then the item can stay within the Testing stage as the Coder works closely with the Testing to diagnose the issue and provide updated code.
If the diagnosing and fixing effort is much larger however, such as requiring more than a day to complete, then the Workitem should move back to the Coding stage even if the WIP limits are exceeded. The priority for the Coder should be to address the defect and get the Workline flowing again in the right direction.
Sometimes defects are found in areas of the code not currently part of a Workitem in progress. These should be brought to the attention of the Product Owner who can create a new Workitem representing the defect and prioritize it accordingly in the Work Backlog.

Estimation

The statistics kept on how long Workitems take to transit the Workline should be enough to estimate  how long any particular item will take to be completed. For example, if the average number of days is  four, then any new item pulled into the Workline is likely to take four days. By the same logic, if there was a set of Workitems that comprised a release, then the time to complete the entire set is four  times the number of items in the set

This estimation technique works well when the scopes of the Workitems are somewhat consistent.  When they are not, some rough estimates of size, combined with the transit time for those specific  sizes can produce a more accurate estimate although it will take longer to gather enough data to be  confident with the results.Be mindful that any additional time spent by the team should be worth the  effort and if not, abandoned for the simple statistics Kanban provides.

PEARL IV : INVEST in good User Stories and SMART Tasks

PEARL IV : INVEST in good User Stories and SMART Tasks

In software development and product management, a user story is one or more sentences in the everyday or business language of the end user or user of a system that captures what a user does or needs to do as part of his or her job function. User stories are used with agile software development methodologies as the basis for defining the functions a business system must provide, and to facilitate requirements management. It captures the ‘who’, ‘what’ and ‘why’ of a requirement in a simple, concise way, often limited in detail by what can be hand-written on a small paper notecard.

User stories originated with Extreme Programming (XP), whose first written description in 1998 only claimed that customers defined project scope “with user stories, which are like use cases”. Rather than offered as a distinct practice, they were described as one of the “game pieces” used in the planning game. However, most of the further literature thrust around all the ways arguing that user stories are “unlike” use cases, in trying to answer in a more practical manner “how requirements are handled” in XP and more generally Agile projects. This drives the emergence, over the years, of a more sophisticated account of user stories.

In 2001, Ron Jeffries proposed the well-known Three C’s formula, i.e. Card, Conversation, Confirmation, to capture the components of a user story:

A Card (or often a Post-it note) is a physical token giving tangible and durable form to what would otherwise only be an abstraction;

A Conversation is taking place at different time and places during a project between the various stakeholders concerned by the given feature (customers, users, developers, testers, etc.), which is largely verbal but most often supplemented by documentation;

The Confirmation, the more formal the better, ensures that the objectives the conversation revolved around have been reached finally.

A pidgin language is a simplified language, usually used for trade, that allows people who can’t communicate in their native language to nonetheless work together. User stories act like this. We don’t expect customers or users to view the system the same way that programmers do; stories act as a pidgin language where both sides can agree enough to work together effectively.

User stories are written by or for business users or customers as a primary way to influence the functionality of the system being developed. User stories may also be written by developers to express non-functional requirements (security, performance, quality, etc.), though primarily it is the task of a product manager to ensure user stories are captured.

Agile projects, especially Scrum , use a product backlog, which is a prioritized list of the functionality to be developed in a product or service. Although product backlog items can be whatever the team desires, user stories have emerged as the best and most popular form of product backlog items.

While a product backlog can be thought of as a replacement for the requirements document of a traditional project, it is important to remember that the written part of an agile user story (“As a user, I want …”) is incomplete until the discussions about that story occur.

It’s often best to think of the written part as a pointer to the real requirement. User stories could point to a diagram depicting a workflow, a spreadsheet showing how to perform a calculation, or any other artifact the product owner or team desires.

When the time comes for creating user stories, one of the developers gets together with a customer representative, e.g. a product manager (or product owner in Scrum), which has the responsibility for formulating the user stories. The developer may use a series of questions to get the customer representative going, such as asking about the desirability of some particular functionality, but must take care not to dominate the idea-creation process.

UserStoryWorkshop

A user story is not a specification, but an communication and collaboration tool. Stories should never be handed off to the development team. They should rather be part of a conversation: The product owner and the team should discuss the stories, or even better, write them together. This leverages the creativity and the knowledge of the team and usually results in better user stories.

As the customer representative conceives a user story, it is written down on a note card (e.g. 3×5 inches or 8×13 cm) with a name and a brief description. If the developer and the customer representative find a user story deficient in some way (too large, complicated, or imprecise), it is rewritten until satisfactory – often using the INVEST guidelines. Commonly, user stories are not to be definite once they have been written down, since requirements tend to change throughout the development lifecycle, which agile processes handles by not carving them in stone upfront.

A Scrum epic is a large user story. There’s no magic threshold at which we call a particular story an epic. It just means “big user story.

“Theme” is a collection of user stories

Important considerations for writing user stories:

  1. Stakeholders write user stories. An important concept is that your project stakeholders write the user stories, not the developers. User stories are simple enough that people can learn to write them in a few minutes, so it makes sense that the domain experts (the stakeholders) write them. As its name suggests, a user story describes how a customer or a user employs the product. You should therefore tell the stories from the user’s perspective.
  2. Personas : A great way to capture your insights about the users and customers is to use personas. But there is more to it: The persona goals help you discover your stories. Simply ask yourself: What functionality does the product have to provide to meet the goal of the personas.A user persona is a representation of the goals and behavior of a hypothesized group of users. In most cases, personas are synthesized from data collected from interviews with users. They are captured in 1–2 page descriptions that include behavior patterns, goals, skills, attitudes, and environment, with a few fictional personal details to make the persona a realistic character. For each product, more than one persona is usually created, but one persona should always be the primary focus for the design.
  3. Use the simplest tool : User stories are often written on index cards as you see in Figure 2(below)  Index cards are very easy to work with and are therefore an inclusive modeling technique.
  4. Remember non-functional requirements: Stories can be used to describe a wide variety of requirements types. For example in Figure 1(below) the Students can purchase parking passes online user story is a usage requirement similar to a use case whereas the Transcripts will be available online via a standard browser is closer to a technical requirement.
  5. Indicate the estimated size: You can see in Figure 2 that it includes an estimate for the effort to implement the user story. One way to estimate is to assign user story points to each card, a relative indication of how long it will take a pair of programmers to implement the story. The team then knows that if it currently takes them on average 2.5 hours per point; therefore the user story in Figure 2 will take around 10 hours to implement.
  6. Indicate the priority: Requirements, including defects identified as part of your independent parallel testing activities or by your operations and support efforts, are prioritized by your project stakeholders (or representatives thereof such as product owners) and added to the stack in the appropriate place. You can easily maintain a stack of prioritized requirements by moving the cards around in the stack as appropriate. You can see that the user story card includes an indication of the priority; Agilists often use a scale of one to ten with one being the highest priority. Other prioritization approaches are possible – priorities of High/Medium/Low are often used instead of numbers and some people will even assign each card it’s own unique priority order number (e.g. 344, 345, …). You want to indicate the priority somehow in case you drop the deck of cards, or if you’re using more sophisticated electronic tooling. Pick a strategy that works well for your team. You also see that the priority changed at some point in the past, this is a normal thing, motivating the team to move the card to another point in the stack. The implication is that your prioritization strategy needs to support this sort of activity. The advice is to keep it simple.
  7. Optionally include a unique identifier: The card also includes a unique identifier for the user story, in this case 173. The only reason to do this would be to do this is if you need to maintain some sort of traceability between the user story and other artifacts, in particular acceptance tests.

Figure 1. Example user stories.

  • Students can purchase monthly parking passes online.
  • Parking passes can be paid via credit cards.
  • Parking passes can be paid via PayPal.
  • Professors can input student marks.
  • Students can obtain their current seminar schedule.
  • Students can order official transcripts.
  • Students can only enroll in seminars for which they have prerequisites.
  • Transcripts will be available online via a standard browser.

Figure 2. User story card (informal, high level).

Format

A team at Connextra developed the traditional user-story template in 2001:

“As a <role>, I want <goal/desire> so that <benefit>”

Mike Cohn, a well-known author on user stories, regards the “so that” clause as optional:

“As a <role>, I want <goal/desire>”

Chris Matts suggested that “hunting the value” was the first step in successfully delivering software, and proposed this alternative as part of Feature Injection:

“In order to <receive benefit> as a <role>, I want <goal/desire>”

Another template based on the Five Ws specifies:

“As <who> <when> <where>, I <what> because <why>.”

User Scenarios

A user scenario expands upon your user stories by including details about how a system might be interpreted, experienced, and used. Like user stories, you might imagine several scenarios for each persona group that you anticipate will make up your audience. Your scenarios should anticipate the user’s goal, specify any assumed knowledge, and speculate on the details of the user’s interaction experience.

User Stories and Planning

There are two areas where user stories affect the planning process on agile projects:

  1. Scheduling. In the agile change management management process where work items, including stories, are addressed in priority order. So, the implication is that the priority assigned to a story affects when the work will be done to implement that requirement. Project stakeholders are responsible for prioritizing requirements.  A numerical prioritization strategy is taken (perhaps on a scale of 1 to 20) and a MoSCoW (Must Should Could Won’t) approach is also used. Stakeholders also have the right to define new requirements, change their minds about existing requirements, and even reprioritize requirements as they see fit. However, stakeholders must also be responsible for making decisions and providing information in a timely manner.
  2. Estimating. Developers are responsible for estimating the effort required to implement the things which they will work on, including stories. The implication is that because you can only do so much work in an iteration, the size of the work items (including stories), affect when those work items will be addressed. Although you may fear that developers don’t have the requisite estimating skills, and this is often true at first, the fact is that it doesn’t take long for people to get pretty good at estimating when they know that they’re going to have to live up to those estimates. If you’ve adopted the pair programming practice then a user story must be able to be implemented by two people in a single iteration/sprint. Therefore if you’re working in one week iterations each user story must describe less than one week worth of work. Of course, if you aren’t taking a non-solo development approach such as pair programming the user story would need to be implementable by a single person within a single iteration. Large stories, sometimes called epics, would need to be broken up into smaller stories to meet this criteria.

INVEST

Bill Wake  introduced the INVEST mnemonic in his seminal post on creating better stories, suggesting they should be Independent, Negotiable, Valuable, Estimable, Small, and Testable. An acute observer of common disconnects between product people and developers, his mnemonic remains an excellent checklist for the creation of high quality product design inputs.

The acronym “INVEST” can remind you that good stories are:

I – Independent
N – Negotiable
V – Valuable
E – Estimable
S – Small
T – Testable

Independent

Ideally, you should be able to implement your stories in any order while still realizing the benefit of seeing something working at the end. This is important for two reasons. First, you lose a lot of the adaptability agile’s supposed to deliver if your stories have (many) interdependencies. Second, it makes it much more likely you’ll end up at the end of an iteration without working product that you can validate to make sure that you’re moving in a valuable direction.

Negotiable… and Negotiated

User stories are not requirements A good story is negotiable. It is not an explicit contract for features; rather, details will be co-created by the customer and programmer during development. A good story captures the essence, not the details. Over time, the card may acquire notes, test ideas, and so on, but we don’t need these to prioritize or schedule stories.They’re not another just another template to put on a piece of paper- they’re part of a process where the product person (original drafter) of the stories co-creates a working incarnation of the story with the implementer. This way, the original author gets someone to help them improve/”edit” their story and make sure the story’s interpreted into product in a way that delivers on something Valuable for the user.

Valuable

A story needs to be valuable. We don’t care about value to just anybody; it needs to be valuable to the customer. Developers may have (legitimate) concerns, but these framed in a way that makes the customer perceive them as important.

This is especially an issue when splitting stories. Think of a whole story as a multi-layer cake, e.g., a network layer, a persistence layer, a logic layer, and a presentation layer. When we split a story, we’re serving up only part of that cake. We want to give the customer the essence of the whole cake, and the best way is to slice vertically through the layers. Developers often have an inclination to work on only one layer at a time (and get it “right”); but a full database layer (for example) has little value to the customer if there’s no presentation layer.

Making each slice valuable to the customer supports XP’s pay-as-you-go attitude toward infrastructure.

Stories want to communication information. Don’t hide them on a network drive, the corporate intranet , or a licensed tool. Make them visible instead, for instance, by putting them up on the wall. A great tool to discover, visualize, and manage your stories is a Product Canvas pasted on the wall.

Estimable

A good story can be estimated. We don’t need an exact estimate, but just enough to help the customer rank and schedule the story’s implementation. Being estimable is partly a function of being negotiated, as it’s hard to estimate a story we don’t understand. It is also a function of size: bigger stories are harder to estimate. Finally, it’s a function of the team: what’s easy to estimate will vary depending on the team’s experience. (Sometimes a team may have to split a story into a (time-boxed) “spike” that will give the team enough information to make a decent estimate, and the rest of the story that will actually implement the desired feature.)

It should be possible for a developer with relevant experience to roughly estimate a story. That estimate’s then used by the product person to prioritize their list of stories, taking account of both value and cost (in terms of developer time).

Small

„ Large stories (epics) are

– hard to estimate
– hard to plan

They don’t fit well into single sprints
„ Compound story

– An epic that comprises multiple shorter stories

„ Complex story

– A story that is inherently large and cannot easily be  disaggregated into constituent stories

Compound stories sometimes hide assumptions, Split compound stories along operational boundaries (CRUD), Data Boundaries.

Good stories tend to be small. Stories typically represent at most a few person-weeks worth of work. (Some teams restrict them to a few person-days of work.) Above this size, and it seems to be too hard to know what’s in the story’s scope. Saying, “it would take me more than a month” often implicitly adds, “as I don’t understand what-all it would entail.” Smaller stories tend to get more accurate estimates.

Story descriptions can be small too (and putting them on an index card helps make that happen). Alistair Cockburn described the cards as tokens promising a future conversation. Remember, the details can be elaborated through conversations with the customer.

Detail can be added to user stories in two ways:

  • By splitting a user story into multiple, smaller user stories.
  • By adding “conditions of satisfaction.”

When a relatively large story is split into multiple, smaller agile user stories, it is natural to assume that detail has been added. After all, more has been written.

The conditions of satisfaction is simply a high-level acceptance test that will be true after the agile user story is complete

Epics are big, coarse-grained user stories. Starting with epics allows you to sketch the product functionality without committing to the details. This is particularly helpful for new products and new features: It allows you to capture the rough scope, and it buys you time to learn more about the users and how to best meet their needs. It also reduces the time and effort required to integrate new insights.

A story map is a graphical, two-dimensional product backlog. At the top of the map are big user stories, which are sometimes “epics” as Mike Cohn describes them, and other times correspond to “themes” or “activities”. These grouping units are created by orienting at the user’s workflow or “the order you’d explain the behavior of the system”. Vertically, below the epics, the actual story cards are allocated and ordered by priority. The first horizontal row is a “walking skeleton” and below that represents increasing sophistication.

In this way it becomes possible to describe even big systems without losing the big picture.

Testable

A good story is testable. Writing a story card carries an implicit promise: “I understand what I want well enough that I could write a test for it.” Several teams have reported that by requiring customer tests before implementing a story, the team is more productive. “Testability” has always been a characteristic of good requirements; actually writing the tests early helps us know whether this goal is met.

Strive for 90% Test Automation.

If a customer doesn’t know how to test something, this may indicate that the story isn’t clear enough, or that it doesn’t reflect something valuable to them, or that the customer just needs help in testing.

A team can treat non-functional requirements (such as performance and usability) as things that need to be tested. Figure out how to operationalize these tests will help the team learn the true needs.

For all these attributes, the feedback cycle of proposing, estimating, and implementing stories will help teach the team what it needs to know.

 Agile User Stories Prioritization 

Aim of Agile User Stories Prioritization  engineering actions are to contribute to business value that is described in terms of return-on investment of software product and it is very  essential for a software. For a product to be  successful, it is very important to identify the  correct equalizer among the competing quality  User Stories. From the customers’ view, the action  of continuous User Stories prioritization creates  the very core of today’s agile process

Continuous requirements prioritization  process from the customer‟s scope of vision forms the  essential of today‟s agile approaches. An essential  feature of any agile approach is an expressed focus on  making business value for the customer . Agile software process practitioners deem this approach  especially valuable for the software producers in a  circumstance that admits extremely uncertain requirements, experimentation with fresh  development technology, and customers willing to  explore the ways in which developing product can  assist their business goals.

The continuous  prioritization of requirements during the project acts  as fundamental role in accomplishing business value  creation. Requirements (re)prioritization at interiteration time is the means to align expert decisions to  the business strategy that aims the business value.

Creating a great user experience (UX) requires more than writing user stories. Also consider the user journeys and interactions, the visual design, and the nonfunctional properties of your product. This results in a holistic description that makes it easier to identify risks and assumptions, and it increases the chances of creating a product with the right user experience.

Agile Development SMART Tasks

A task communicates the need to do some work. Each team member can define tasks to represent the work that they need to accomplish. For example, a developer can define development tasks to implement user stories. A tester can define test tasks to assign the job of writing and running test cases. A task can also be used to signal regressions or to suggest that exploratory testing should be performed. Also, a team member can define a task to represent generic work for the project.

You link tasks to a user story to track the progress of work that has occurred to complete that story. After you define a task, you can link it to the user story that it implements

During the Iteration Planning meeting, stories and sometimes defects are broken down into tasks. The tasks are all the work the team must complete to develop, test and accept the story as done.  Tasks typically range in size from 1 hour to 2 days, depending on the length of the iteration. Two-week iterations ideally have tasks of 8 hours or less, and four week iterations have tasks sized at 2 days or less. Tasks larger than these guidelines should be broken down further to allow the team to incrementally complete the work and show progress. Keep tasks small. Estimate all tasks in hours. Iteration tracking will be done at story level to keep the team focused on business value.

The entire team works together to come up with the tasks for all the stories identified for the iteration. The majority of the tasks are determined during the iteration planning meeting, but in some cases the team may discover tasks as the work starts. It is appropriate to add and/or remove tasks on stories during the iteration.

When a task involves an external group – perhaps you want to demo your system to some project stakeholders, or deploy a testable version into your QA/test environment – then you should consider including the task as a reminder to coordinate with those groups.

Tasks may include the traditional steps in a development lifecycle (although limited to the feature in question, not the entire product). For instance: Design, Development, Unit Testing, System Testing, UAT (User Acceptance Testing), Documentation, etc. Remember, agile software development methods do not exclude these steps. Agile methods just advocate doing the steps feature-by-feature, just in time, instead of in big phases.

At the end of the day , the individuals update the tasks hours remaining for the tasks they have taken responsibility . This will enable to determine the Burn down chart.

Tasks makes the micro work break down structure that the teams can use to facilitate coordinating, estimating, tracking status and and assigning individual responsibilities to help assure completion of the stories. and there by the iteration

There is an acronym for creating effective goals: “SMART” –

S – Specific
M – Measurable
A – Achievable
R – Relevant
T – Time-boxed
These are good characteristics for tasks as well.

Specific

A task needs to be specific enough that everyone can understand what’s involved in it. This helps keep other tasks from overlapping, and helps people understand whether the tasks add up to the full story.

Measurable

The key measure is, “can we mark it as done?” The team needs to agree on what that means, but it should include “does what it is intended to,” “tests are included,” and “the code has been refactored.”

Achievable

The task owner should expect to be able to achieve a task. XP teams have a rule that anybody can ask for help whenever they need it; this certainly includes ensuring that task owners are up to the job.

Relevant

Every task should be relevant, contributing to the story at hand. Stories are broken into tasks for the benefit of developers, but a customer should still be able to expect that every task can be explained and justified.

Time-Boxed

A task should be time-boxed: limited to a specific duration. This doesn’t need to be a formal estimate in hours or days, but there should be an expectation so people know when they should seek help. If a task is harder than expected, the team needs to know it must split the task, change players, or do something to help the task (and story) get done.

As you discuss stories, write cards, and split stories, the INVEST acronym can help remind you of characteristics of good stories. When creating a task plan, applying the SMART acronym can improve your tasks.

Scrum and Sprints

The task board is the single most important information radiator that an agile team has. A task board illustrates the progress that an agile team is making in achieving their sprint goals. Usually the task board is located in an area that is central to the team

All of these task boards have a few simple things in common:

1) 4 basic columns (there can be more)

  1. Stories
  2. To Do
  3. In Process (In progress, WIP, etc.)
  4. Complete (Completed, Done, etc.)

2) A Row for each story

Simple task board

Taskboard

More complicated version that looks something like above.

PEARL I : SCRUM : An Introduction on Origin,Evolution,Definition,Adoption and Future

PEARL I : SCRUM : An Introduction on Origin,Evolution,Definition,Adoption and Future

Origin

The first software development Scrum was created at Easel Corporation  in 1993 based on extensive research on successful projects worldwide, a deep analysis of the computer science literature, close collaboration with leading productivity experts, and decades of experience with advanced software technologies. Jeff Sutherland was the Chief Engineer for the Object Studio team that defined roles, hired the first Product Owner and ScrumMaster, developed the first Product Backlog and Sprint Backlog and built the first portfolio of products created with Scrum. They built the first object-oriented design and analysis tool that  incorporated round-trip engineering in the initial SCRUM. A second SCRUM  implemented the first product to completely automate object-relational mapping in an  enterprise development environment. Jeff was assisted by two world-class developers, Jeff  McKenna, now an extreme programming (XP) consultant, and John Scumniotales, now a  development leader for object-oriented design tools at Rational Corporation.

There were some key factors that influenced the introduction of SCRUM at Easel Corporation. The book “Wicked Problems, Righteous Solutions” (DeGrace and Stahl 1990) reviewed the reasons why the waterfall approach to software development does not work for software development today. Requirements are not fully understood before the  project begins. The user knows what they want only after they see an initial version of the  software. Requirements change during the software construction process. And new tools  and technologies make implementation strategies unpredictable. DeGrace and Stahl  reviewed “All-at-Once” models of software development which uniquely fit object oriented implementation of software.
In 1995, Jeff introduced the Scrum team to Ken Schwaber, CEO of Advanced Development Methods. Ken agreed that Scrum was a better way to build software and they worked together to formalize the Scrum development process at OOPSLA’95 . In the same year, Sutherland provided support for development of eXtreme Programming by giving Kent Beck all background information on the creation of Scrum and the results of two years of product development with the Scrum process from 1993-95. XP engineering practices evolved along with Scrum and the two leading Agile development processes work well together. Scrum and XP are the most widely used Agile processes worldwide and their creators are signatories of the Agile Manifesto.

One of the influences that sparked the creation of the Scrum Agile development process was a Harvard Business Review paper on Japanese new product  development by Takeuchi and Nonaka  . A key  component of their presentation was a chart showing product development separated into silo’s (Type A),  phases slightly overlapped (Type B), and all phases of  development overlapping (Type C). The Authors viewed  Type A product development as implemented at NASA as  an outmoded relay race process. Fuji-Xerox abandoned the  NASA approach for Type B which they called “Sashimi”  because slices of work overlapped with collaboration  between phases. Type C was implemented at Canon and  Honda. Takeuchi and Nonaka envisioned Types B and C as  a Rugby approach where multiple phases of product  development are done simultaneously. Scrum is a Rugby  formation and they viewed an “all-at-once” process as  similar to a Rugby team moving down the field passing the  ball back and forth

Jeff Sutherland got the name “Scrum” from the famous article “The New New Product Development Game,” which was published by Takeuchi and Nonaka in the Harvard Business Review in 1986. This article describes how companies like Honda, Cannon, Fuji-Xerox produced world class results using a scalable, team based approach to all at once product development. It also emphasizes the importance of empowered self organizing teams and outlines managements role in the development process.

On the first page of the article they write: “Companies are increasingly realizing that the old, sequential approach to developing new products simply won’t get the job done. Instead companies

in Japan and the United States are using a holistic method – as in rugby, the ball gets passed within the team as it moves as a unit up the field.

The Evolution

Evolution occurs in dynamic response to environmental demands.

After discussing the notion of various types of Scrum  with Certified ScrumMasters in Scrum Alliance workshops  and with development teams at Microsoft, Yahoo, Ariba,  Adobe, GE Healthcare, and other companies, it appeared  that the chart in article can be applied to a higher level of  thinking about three types of Scrum—going beyond the  thinking of Takeuchi and Nonaka

In a Type A Scrum, all development occurs in an increment within the time box of Scrum iteration called a Sprint. A side effect of this approach is downtime between iterations when reorganizing for the next Sprint. Nevertheless, well executed Sprints can double productivity and repeatedly deliver projects on time, within budget, with functionality precisely targeted to end-user demands.
By adding product definition tasks for the next Sprint into the current Sprint, a Type B Sprint allows work to flow smoothly from Sprint to Sprint. Product backlog requirements for the next Sprint are developed in the current Sprint. This has enabled some development organizations to deliver more working product than sales, marketing, or customers can absorb. The development bottleneck is eliminated and the company can adopt new strategies and create new products that were previously impossible to deliver.
Type C Sprint can be envisioned as overlapping Sprints by running software releases through the same Scrum team at the same time. This requires experienced Scrum teams, well designed product architecture, and automation of Product and Sprint backlogs. Throughput can be enhanced to deliver dozens of new releases of enterprise software annually. Competitors can be overwhelmed and market dominance achieved.

Type C Scrum increases speed of development,  aligns individual and corporate objectives, creates a  culture driven by performance, supports shareholder  value creation, achieves stable and consistent  communication of performance at all levels, and enhances  individual development and quality of life

To deal with simultaneous weekly and monthly  Sprints, along with quarterly major releases, a MetaScrum  was formed at PatientKeeper led by the lead Product  Owner. This is a weekly meeting that typical takes 1.5  hours and includes the CEO and other senior  management, as well as leadership from marketing,  development, quality assurance, installation, and support.
Takeuchi and Nonaka observed that collapsing phases of product development improved innovation, throughput, time to market, and product acceptance. As market pressures have evolved and changed, it is possible to collapse Scrum Sprints to create a dramatic increase in business opportunity.

Sutherland and Schwaber evolved  Scrum through trial and error at a number of companies during the  s – NewsPage, IDX Systems, Fidelity Investments, and PatientKeeper.  The Daily Scrum, the burndowns and  the Sprint Backlog were added, and  planning Sprints were dropped. !e  importance of self-managing, cross functional teams became apparent and a bedrock to Scrum.

Schwaber also collaborated with process control scientists at DuPont to tie Scrum to first principles in industrial process control theory, learning that the reason Scrum worked was
that it was empirical.

“In , the Agile Manifesto was formed,” says Schwaber. “!e signatories gathered in SnowBird because they felt their approach to software development had  something in common that was far better and more people-oriented than the  emerging favored process of the day, Rational Unified Process. !e principles  of the manifesto are the common touch
points that we agreed upon.”
Today, Schwaber notices that the term “Agile” is often used in a careless way.“Many people think that there are Agile processes. There aren’t. There are processes that conform to the Agile  Manifesto principles, such as Scrum and  Extreme Programming. However, the  word Agile is often used as a description  for ‘we aren’t using waterfall.’ That is not what Agile means!”
In Schwaber’s mind, Scrum and Agile  started emerging for two reasons: The Agile Manifesto was a rallying point for dissatisfaction and a new  approach. The emergence of new IDEs  that were as powerful as SmallTalk, like VisualStudio and Eclipse, were the technological enablers.

Scrum has streamlined software development — and professionals from around the world are starting to see the value of using Scrum. Of all the possible Agile frameworks used by companies, 66 percent are Scrum or Scrum variants.

Scrum has the power to transform project management across every industry, every business, and even across life in general. By using Scrum, teams become more Agile, discovering how to react more quickly and respond more accurately to the inevitable change that comes in the teams way. And by staying focused, collaborating, and communicating, you can accomplish what truly needs to be done — successfully.

Most important, Scrum is not unproven hype. It’s a solid and successful Agile framework that’s been applied to a variety of projects and teams. Universities use Scrum to deliver valued projects to clients. Militaries have relied on Scrum to prepare ships for deployment. In the automotive world, Team Wikispeed is using Scrum to build a fast, affordable, ultra-efficient, safe commuter car that should sell for less than $20,000.

So whether you’re working on the next smartphone app, managing logistics for a store, or planning a charity event, you should take a closer look at using Scrum. And Scrum Alliance can give you the proven framework, best implementation practices, and supportive guidance you need to achieve success.

The most profitable software product ever created (Google Adwords ) is powered by Scrum and the most productive large project with over a million lines of code (SirsiDynix and Starsoft) used a distributed, outsourced, Scrum implementation. CMMI Level 5 companies cut costs in half with Scrum while simultaneously improving quality, customer satisfaction, and the developer experience (Systematic Software Engineering). At the same time, Scrum remains the process of choice in small entrepreneurial companies where it has it roots.

Scrum Theory
Scrum is founded on empirical process control theory, or empiricism. Empiricism asserts that  knowledge comes from experience and making decisions based on what is known. Scrum employs an iterative, incremental approach to optimize predictability and control risk.  Three pillars uphold every implementation of empirical process control: transparency,  inspection, and adaptation.

Transparency
Significant aspects of the process must be visible to those responsible for the outcome.
Transparency requires those aspects be defined by a common standard so observers share a  common understanding of what is being seen.
For example:

  • A common language referring to the process must be shared by all participants; and,
  • Those performing the work and those accepting the work product must share a common definition of “Done”

Inspection
Scrum users must frequently inspect Scrum artifacts and progress toward a Sprint Goal to detect  undesirable variances. Their inspection should not be so frequent that inspection gets in the way  of the work. Inspections are most beneficial when diligently performed by skilled inspectors at  the point of work.
Adaptation
If an inspector determines that one or more aspects of a process deviate outside acceptable  limits, and that the resulting product will be unacceptable, the process or the material being  processed must be adjusted. An adjustment must be made as soon as possible to minimize  further deviation.
Scrum prescribes four formal events for inspection and adaptation, as described in the Scrum

  •  Sprint Planning
  •  Daily Scrum
  •  Sprint Review
  •  Sprint Retrospective

Definition

Scrum is based on small set of values, principles, and practices called collectively the Scrum Framework. Organizations using scrum shall use the scrum framework entirely perhaps not through the entire organization at once, but certainly within the initial teams that use scrum

Core Scrum — Values and roles

Values from the Agile Manifesto

Scrum is the best known of the Agile frameworks. It is the source of much of the thinking behind the values and principles of the Agile Manifesto, which forms a common basis for all of these approaches.

The Agile Manifesto values apply directly to Scrum:

  • Individuals and interactions over processes and tools. Scrum, like all the Agile frameworks and methods, relies directly on trust in teams, the individuals in the teams, and the way they interact. Teams figure out what is to be done, teams figure out how to do it, and teams do it. Teams identify what’s getting in their way, and they take the responsibility to resolve all the difficulties that are within their scope. Teams work with other parts of the organization to resolve the concerns that are outside their control. This is critical. Trying to do Scrum but undermining this primary focus on team responsibility and autonomy will generally lead to trouble.
  • Working software over comprehensive documentation. Scrum requires a working, finished increment of the product as the primary result of every sprint. Certainly there will be analysis work, design work, and testing work, all of which may need to be documented. But it is having a result (working software) that allows the organization to guide the project to success. This is critical. Scrum teams must produce a product increment in every sprint.
  • Customer collaboration over contract negotiation. The Scrum product owner is the Scrum team’s prime point of contact with the eventual end users of the product. The product owner is a member of the team and works collaboratively with the team to determine what needs to be done. In this collaboration, the product owner selects the most valuable next things to do, ensuring that the product has the highest possible value at every point in time. This is critical. The product owner needs to build a rich collaboration with the team.
  • Responding to change over following a plan. Everything about Scrum is designed to make sure that everyone has the information they need to make good decisions about the project. The project’s progress is represented by a real, running product increment. The backlog of things to be done is available for all to see. Progress, both overall and sprint by sprint, is clearly visible. Problems and concerns are discussed openly and dealt with immediately. This is critical. Scrum works well for teams that openly inspect what’s going on and adapt their actions to the reality. It works poorly for those who do not.

Scrum values

All work performed in Scrum needs a firm foundation of values for the team’s process and principles. With its emphasis on teamwork and continuous improvement, Scrum both creates those values and relies on them. The values are focus, courage, openness, commitment, and respect. From the team’s perspective, the values offer the following guides:

  • Focus. Because we focus on only a few things at a time, we work well together and produce excellent work. We deliver valuable items sooner.
  • Courage. Because we are not alone, we feel supported and have more resources at our disposal. This gives us the courage to undertake greater challenges.
  • Openness. As we work together, we practice expressing how we’re doing and what’s in our way. We learn that it is good to express concerns so that they can be addressed.
  • Commitment. Because we have great control over our own destiny, we become more committed to success.
  • Respect. As we work together, sharing successes and failures, we come to respect each other and to help each other become worthy of respect.

If an organization will let Scrum do its work, everyone involved will discover the benefits and will begin to understand why Scrum both engenders and relies upon these values.

The Scrum framework

Scrum is a framework for building a product.

Scrum is also a team process that begins when stakeholders need a product. The Scrum team includes three roles: the product owner, the ScrumMaster, and the members of the development team. These roles are further described below. The product is built incrementally over a series of short time periods called sprints. A sprint is a fixed time period, up to four weeks long but with a preference toward shorter intervals. During each sprint, the Scrum team builds and delivers a product increment. Each increment is a recognizable, visibly improved, operating subset of the product, meeting understood acceptance criteria and built to a level of quality referred to as the Definition of Done.

Scrum includes three essential artifacts: the product backlog, the sprint backlog, and the product increment. The product backlog is the list of ideas for the product, in the order we expect to build them. The sprint backlog is the detailed plan for development during the next sprint. The product increment, a required result of every sprint, is an integrated version of the product, kept at high enough quality to be shippable if the product owner chooses to ship it.

In addition, Scrum requires transparency within the team and with the stakeholders. Therefore the Scrum team produces visible displays of plans and progress.

Scrum also includes five activities or meetings. These are the backlog refinement, sprint planning, daily scrum, sprint review, and sprint retrospective.

SCRUM project planning uses lightweight techniques such as Burndown  Charts, as opposed to PERT charts. A PERT chart is only as good as the assumptions  inherent in the critical path represented on the chart. In agile development, the critical  path usually changes daily, rendering any given PERT chart obsolete within 24 hours.  The solution is using a technique to calculate the velocity of development. The neural  networks in the brains of team members are used on a daily basis to calculate the critical  path. This allows the plan to be recalculated and the velocity of “burndown” of work to  be computed. Team efforts to accelerate or decelerate the velocity of burndown allow a  team to “fly” the project into a fixed delivery date.

SCRUM is  the only agile methodology that has been formalized and published as an organizational  pattern for software development (Beedle, Devos et al. 1999). The process assumes that  requirements will change during the period between initial specification and delivery of a  product. It supports Humphrey’s Requirements Uncertainty Principle which states that for  a new software system, the requirements will not be completely known until after the  users have used it. SCRUM allows for Ziv’s Uncertainty Principle in software  engineering – uncertainty is inherent and inevitable in software development processes  and products (Ziv and Richardson 1997). And it support Wegner’s Lemma which states  that it is not possible to completely specify an interactive system (Wegner 1995). Most  software systems built today are object-oriented implementations which depend on  environmental inputs to determine process outputs, i.e. interactive systems.

Scrum roles

Product owner

The product owner has responsibility for deciding what work will be done. This is the single individual who is responsible for bringing forward the most valuable product possible by the desired date. The product owner does this by managing the flow of work to the team, selecting and refining items from the product backlog. The product owner maintains the product backlog and ensures that everyone knows what is on it and what the priorities are. The product owner may be supported by other individuals but must be a single person.

Certainly the product owner is not solely responsible for everything. The entire Scrum team is responsible for being as productive as possible, for improving its practices, for asking the right questions, for helping the product owner.

Nonetheless, the product owner, in Scrum, is in a unique position. The product owner is typically the individual closest to the “business side” of the project. The product owner is charged by the organization to “get this product out” and is the person who is expected to do the best possible job of satisfying all the stakeholders. The product owner does this by managing the product backlog and by ensuring that the backlog, and progress against it, is kept visible.

The product owner, by choosing what the development team should do next and what to defer, makes the scope-versus-schedule decisions that should lead to the best possible product.

Development team member

The development team is made up of the professionals who do the work of delivering the product increment. Scrum requires that the development team be a cross-functional group of people who, as a group, have all the necessary skills to deliver each increment of the product.

The development team members are expected to be available to the project full time, not splitting their time over numerous projects. They have the responsibility of self-organizing to accomplish each sprint goal, producing each new product increment according to each sprint plan.

The product owner makes an ordered list of what needs to be done; the development team members forecast how much they can do in one sprint, and they decide how they are going to do it.

ScrumMaster

The ScrumMaster is a “servant leader” who helps the rest of the Scrum team follow the process. The ScrumMaster must have a good understanding of the Scrum framework and the ability to train others in its subtleties.

The ScrumMaster helps the product owner understand how to create and maintain the product backlog. He or she works with the entire Scrum team to evolve the Definition of Done. The ScrumMaster also works with the development team to find and implement the technical practices needed to get to Done at the end of each sprint.

Another responsibility of the ScrumMaster is to remove impediments to the team’s progress. These impediments may be external to the team (such as a lack of support from another team) or internal (such as the product owner not knowing how to properly prepare the product backlog). That said, the ScrumMaster fosters self-organization, meaning that the team itself should remove issues wherever possible.

The ScrumMaster may facilitate meetings and always acts as a coach for the Scrum team, helping it execute the Scrum process. He or she helps team members work together and learn the Scrum framework, and protects them from both internal and external distractions. The ScrumMaster keeps the Scrum team on track, productive, and growing in ability.

Ultimately, the ScrumMaster is responsible for ensuring that Scrum is understood and in place, inside the team and outside. He or she helps people outside the team understand the process and the kinds of interactions with the team that are helpful (and those that are not). The ScrumMaster helps everyone improve to make the Scrum team more productive and valuable.

Artifact: Product backlog

The product backlog is crucial tool that enables the scrum team to achieve fast, flexible, value delivery flow in the presence of uncertainty.

The product backlog is an essential artifact in Scrum. It is an ordered list of ideas for the product, kept in the order in which the team expects to do them. It is the single source from which all requirements flow. This means that all the work the development team does comes from the product backlog. Every feature idea, enhancement, bug fix, documentation requirement — every bit of work the team does — is derived from a product backlog item. Each item on the product backlog includes a description and an estimate of the team capacity required to produce it.

Scrum Overview

The product backlog may begin as a large list or a short one. It may be vague or detailed. Typically, it begins short and vague and becomes longer and more concrete as time goes on. Product backlog items slated for implementation soon will be clarified, better defined, and split into smaller chunks as part of the product backlog refinement activity.

The Product Owner is accountable for maintaining the product backlog, although he or she may — and should — have help in producing it and keeping it up to date. Individual product backlog items may originate from the Product Owner, from team members, or from other stakeholders.

Activity: Product backlog refinement

Since product backlog items are often large and general in nature, and since ideas come and go and priorities change, product backlog refinement is an ongoing activity throughout a Scrum project. This activity includes but is not limited to:

  • Keeping the product backlog ordered
  • Removing or demoting items that no longer seem important
  • Adding or promoting items that arise or become more important
  • Splitting items into smaller items
  • Merging items into larger items
  • Estimating items

One key benefit of the product backlog refinement activity is that it helps prepare for upcoming sprints. Considerations include but are not limited to:

  • Whether each item entering the sprint truly represents an increment of “business value”
  • Whether the development team needs to be able to build each item within just a single sprint
  • Whether everyone on the team has a clear understanding of the purpose of each item

Depending on the nature of the product, other skills and inputs may be necessary. In every case, product backlog refinement is best considered as an activity for all team members, not just for the Product Owner.

Activity: Sprint planning

Each sprint begins with a time-boxed meeting called the sprint planning meeting. In this meeting the Scrum team collaborates to select and understand the work to be done in the upcoming sprint.

The entire team attends the sprint planning meeting. Working from the ordered product backlog, the Product Owner and the development team members discuss each item and come to a shared understanding of that item and what is required to complete it according to the current Definition of Done. All Scrum meetings are time-boxed. The recommended time for the sprint planning meeting is two hours or less per week of sprint duration. Because the meeting is time-boxed, the success of the sprint planning meeting is highly dependent upon the quality of the product backlog going in. This is why product backlog refinement is an important Scrum activity.

Scrum moves control from a central scheduling dispatching authority to individual teams doing the work.

In Scrum, the sprint planning meeting is described as having two parts:

  1. Determine what work will be completed in the sprint
  2. Determine how the work will be accomplished

In the first part of the meeting, the Product Owner presents ordered product backlog items to the development team, and the whole Scrum team collaborates to understand the work.

The number of product backlog items to undertake in the sprint is solely up to the development team. To decide how many items to undertake, the development team considers the current state of the product increment, the past performance of the team, the team’s current capacity, and the ordered product backlog. The development team alone decides how much work to take on. Neither the Product Owner nor any other agency can push more work onto the development team.

Often, but not always, the sprint has a goal. Having a sprint goal is a strong practice that helps everyone focus more on the essence of what needs to be done and less on small details that may not truly be important for the final product.

In the second part of the meeting, the development team collaborates to decide how to produce the next product increment in accordance with the current Definition of Done. The team does sufficient design and planning to be confident of completing the work during the sprint. Work to be done in the early days is broken down into small units of one day or less. Work to be done later may be left in larger units to be decomposed later.

Deciding how to do the work is the responsibility of the development team, just as deciding what to do is the responsibility of the Product Owner.

The Product Owner needs to be readily available to the team even if he or she does not attend this part of the meeting. However, the Product Owner may remain to answer questions and resolve misunderstandings.

Sprint planning concludes with the Scrum team reaching a common understanding of the quantity and complexity of what is to be accomplished during the sprint and with a rational expectation of being able to complete it. The development team forecasts the amount of work it will complete and commits to accomplishing it.

To summarize, in the sprint planning meeting the development team:

  • Considers and discusses product backlog items with the Product Owner
  • Ensures that all team members understand those items
  • Selects a number of items to achieve
  • Creates a sufficiently detailed plan to ensure that achievement

The resulting list of things to do is the sprint backlog.

Artifact: Sprint backlog

The sprint backlog is the list of refined product backlog items chosen for development in the current sprint, together with the team’s plan for accomplishing that work. It reflects the team’s forecast of what work can be completed.

With the sprint backlog in place, the sprint begins, and the development team develops the new product increment defined by the sprint backlog.

Development

During the sprint, the development team self-organizes to produce the defined product increment. Self-organizing means that the development team determines how to produce and then does produce the product increment, in accordance with organizational standards, according to the Definition of Done.

Artifact: Product increment

The most important Scrum artifact is the product increment. Every sprint produces a product increment that must be of high enough quality to be given to users. The product increment must meet the Scrum team’s current Definition of Done, and each component of it must be acceptable to the Product Owner.

Additional indicators of visible progress

Scrum requires transparency within the team and outside the team. While the product increment is the most important way of creating transparency, the Scrum team will create any other artifacts that are needed to make sure that the status of the project is understood. Such artifacts commonly include burn charts and task boards.

Definition of Done

When the product increment is delivered, it needs to be “done” according to a shared understanding of what “done” means. This definition is different for every Scrum team, and as the team matures the Definition of Done will expand and become more stringent.

The Definition of Done must always include the notion that the product increment be of high enough quality to be shippable: The Product Owner could choose to release it immediately. The product increment includes the functionality of all previous product increments and is fully tested so that all completed product backlog items continue to work together.

Activity: Daily Scrum

The self-organizing development team uses the daily Scrum meeting to ensure that it is on track for attaining the sprint goal. The meeting takes place at the same time and place, every day. Each development team member gives three pieces of information:

  • What I have accomplished since our last daily Scrum
  • What I plan to accomplish between now and our next daily Scrum
  • What (if anything) is impeding my progress

There may be brief, clarifying questions and answers, but there is no discussion of any of these topics during the daily Scrum itself. However, many teams meet right after the daily Scrum to work on any issues that have come up.

The daily Scrum is not a report to management, nor to the Product Owner, nor to the ScrumMaster. It is a communication meeting within the development team that ensures that team members are all on the same page. Only the Scrum team members, including ScrumMaster and Product Owner, speak during this meeting (although other interested parties may come and listen). Based on what comes up in the meeting, the development team reorganizes the work as needed to accomplish the sprint goal.

The daily Scrum is a key element of Scrum, leading to transparency, trust, and better performance. It provides rapid recognition of problems and builds the team’s self-organization and self-reliance. All Scrum meetings are timeboxed; the recommended timebox for the daily Scrum is no more than 15 minutes.

Activity: Sprint review

At the end of the sprint, the Scrum team and stakeholders review the sprint’s output. The recommended timebox for the sprint review is one hour per week of sprint duration.

The central point of discussion is the product increment completed during the sprint. It is generally both wise and helpful for the stakeholders to attend this meeting, since they are directly interested in the product. This is an informal meeting to look at where the team is and to collaborate on how it might go forward. Everyone has input at the sprint review. Naturally, the Product Owner makes the final decisions about what happens next, updating the product backlog as appropriate.

Every teams will find its own way to handle the sprint review. A demonstration of the product increment is a common procedure. Groups also often discuss what they observed during the sprint and what product ideas came to mind. They discuss the state of the product backlog and talk about possible completion dates and what might be achieved by those dates.

The sprint review gives everyone present an overview of the current product increment; therefore, it is common to update the product backlog as part of the sprint review.

Activity: Sprint retrospective

The Scrum team improves its own process, always remaining within the Scrum framework. Therefore, at the end of each sprint, the Scrum team meets for the sprint retrospective. The purpose is to review how things went in terms of the process, relationships among people, and the tools. The team identifies what went well and what didn’t go so well, and it identifies potential improvements. The recommended timebox for the sprint retrospective is one hour per week of sprint duration.

Rinse, repeat

This Scrum cycle repeats for every sprint.

To sum up, the Scrum team’s members (the Product Owner, the development team, and the ScrumMaster) collaborate to create a series of product increments during short, time-boxed intervals called sprints. Each increment meets the Product Owner’s acceptance criteria and the team’s shared Definition of Done. The team works from a product backlog. During each sprint, the team begins with sprint planning to produce the sprint backlog, which is a plan for the sprint. The team self-organizes to handle the development, using daily Scrum meetings to coordinate and to ensure that they are producing the best possible product increment. The team performs product backlog refinement to prepare for the next sprint’s planning meeting. It ends the sprint with the sprint review and sprint retrospective, reviewing the product and the process.

Adoption and Future

“Scrum and Agile are similar to Lean Thinking. Scrum is a new way of doing business, but there won’t be that many that reap all of its benefits – such as productivity,  quality, engaged people, and maximized return on investment. American automobile companies were immersed  in running their operations top-down, using the principles of Scientific Management to try to plan everything.
Even while Toyota began to reach their market by using flexible lean techniques, they were unable to adapt. The change in perceptions, culture, and habits was too great.  Many places that are trying to shift from the predictive  approaches of waterfall and command and control are  having the same problems. The change in thinking is  particularly hard. The improved engineering practices  to support the change are also hard. Ken Schwaber predicted that only  those organizations with compelling reason to change  and management with insight and courage will succeed.”
“John Chambers, the CEO of Cisco, put his organization through such a change over the last four years and commented that it was the hardest thing he had ever  done.

“I predict that only 15–25 % of all organizations attempting to use Scrum throughout the enterprise will  succeed in making the change and reaping the benefits. As a result, they will out-compete those who weren’t  able to change and become dominant players in their markets”

In February and March of 2013, Scrum Alliance,  ProjectManagement.com and  ProjectsAtWork surveyed their  readership on use, knowledge  and views of Scrum.  The extensive survey of nearly 500 participants included  in-depth interviews and a literature review.  Participants from more than 70 countries responded to the  survey, with the United States leading at 35% and India second at 12%. The United Kingdom,Canada and Australia accounted  for 4% each, with Germany at  3% and South Korea, Mexico,Belgium, Brazil, Malaysia and New Zealand comprising 2%.

The majority of the participants are from the IT industry (41%),but a good representation camefrom Finance (12%), Government(6%), Healthcare (6%) and Telecommunications (5%), indicating the growing interest in Scrum outside IT. 53% of the participants were from organizations with 1,000 or fewer employees, yet 44% were with companies with annual revenues of $50M to $1B+. 28% were from organizations with ISO 9001 certification, but nearly half (48%) did not have any.

Key Findings
• While 41% of organizations are jumping into Agile waters without requiring a specific certification of their employees, 54% either agree or strongly agree that a certification such as the Certified Scrum Professional (CSP) improves their chances of sustained success.
• Culture is king in the Agile world—and according to a majority of respondents, organizations must create cultures that encourage collaboration in order to deliver value to their customers. This includes fostering self-organized teams and active support from management. Scrum facilitates all of these success factors.
• Scrum is the overwhelmingly preferred Agile method, used by 40% of respondents.The second-most popular method is Kanban, often using many elements of Scrum,a noteworthy trend as organizations seek to find for themselves what works for their specific domains and needs.
How Often and Why They Use Scrum
• In terms of current Agile approaches, Scrum leads the way: 40% of those sampled claimed to be adherents. It was followed by Kanban (15%) and Lean (11%). 19% of the participants used Scrum for up to a quarter of their projects that fall outside of IT, the
majority (38%) of which are in R&D, operations or production.
• 60% of the survey participants used Scrum regularly. 39% used Scrum more broadly throughout their business as one of their project management practices, while 16% used it exclusively for software development projects. 46% of the participants are deploying and managing Scrum projects within a Project Management Office (PMO), and 24% feel that managing and deploying Scrum projects this way is effective and successful.
• In terms of business priorities for Scrum projects, 41% feel that fulfilling customer needs is highest, while meeting budget, time and scope constraints as well as engaging in projects that drive innovation and market share followed with 36%. In terms of providing customer satisfaction while using Scrum, 27% feel having active senior management support is vital,  and 22% say a clear set of business goals for what gets achieved is necessary. That is why over half felt that cultural influences such as an open and collaborative environment—and empowered/self-organizing Scrum teams—are vital for facilitating Scrum; that can only be achieved through active support of management and clear business vision.

Scrum Adoption Factors
• The common belief that the vast majority of organizations adopting and integrating Scrum come from a Waterfall-like background was supported by the survey. 24% use Scrum for some projects while using Waterfall for the rest. 13% use Scrum exclusively, while 31% indicate no use of Waterfall. 23% felt it was a difficult transition from a Waterfall-based method to Scrum. Another perception, stated by 25% of respondents, was that there were no clearly identified metrics to identify and measure the successful delivery of Scrum projects.

Scrum will continue to grow because it is poised for the “Age  of the Customer.”What lies at the  heart of Agile is the  obsessive focus on  service to the  customer—the  mainstay of almost  any business.

Scrum will continue to expand outside of software development. Another adoption of Agile
outside of software development  presents itself in the article “Agile  Reinvents Retail,” published by  John Hitchcock of SandHill.com  on September 2012. The story  describes how a California retail  outlet called Oddyssea used Agile development principles to deploy  its retail operations

Another recent and famous  example of Scrum outside  of software development is  embodied in a sports car called  Wikispeed. Built by Joe Justice, a software engineer by trade, and  a team who entered the 2008  X-Prize competition, Wikispeed  was deployed using Scrum and  crowdsourcing. The team was able  to place tenth in a crowded and  highly competitive environment.
This was done by getting a working  prototype in three months for a  car that can go from 0 to 60 mph  in less than five seconds, weighs  just 1,404 pounds, has a top speed  of 149 mph and gets more than  100 mpg—all using Agile practices  and techniques.

Scrum is well suited for complex problems, where routine cookie cutter solutions does not apply.  In complex domains the ability to probe, explore, sense,inspect, adapt,and respond is critical.

PEARL VIII : Agile Test Automation : Application of agile development principles to Test Automation

PEARL VIII :  Agile Test Automation : Application of agile development principles to Test Automation

Introduction

Today, more and more software development companies are shifting to an agile development approach from traditional waterfall development to keep pace with the current trends.To meet the demands of agile environment, several companies have adopted test automation which resolves some of the issues faced in manual testing and delivers faster results.

Even though test automation provides assistance in many ways, there are several challenges in test automation which can turn out to be a nightmare if it is adopted without appropriate brainstorming and analysis. Here’s a complete list of approaches to make test automation work successfully in agile environment.

The Development methodology agile team follows when creating automated tests is very similar to the process they follow for creating the software being tested. It involves a fair bit of design, coding and testing of its own to get it working correctly. So just like the application itself, automated tests are best developed incrementally – adding new tests and features to the automation framework over several different sprints. It’s important not to aim for the perfect “it-can-do-everything” test framework right at the start, as it would never materialize. Balance the cost vs. ROI well and come up with a bare minimum working solution at the start.

As more companies move from a traditional waterfall software development approach to an agile Methodology, suppliers are offering more test automation tools and services. Automated tests can provide faster feedback than a manual test, reducing rework and long feedback cycles.

Manual testing, particularly manual exploratory testing, is still important. However,  Agile teams typically find that the fast feedback afforded by automated regression is  a key to detecting problems quickly, thus reducing risk and rework.

Research firm Forrester expects testing-as-a-service (TaaS) to emerge as a managed test service from an increase in test automation, as large suppliers such as HP seek to support customers with test automation tools and outsourcing options, including security testing and services for SAP. Better test automation is intrinsic to agile development

Working tests, just like working software, are useful; they build confidence and get everyone excited about the progress being made. Successes, even small ones, make it easier to bring everyone on-board – especially when the test automation solution team have created actually runs and proves to be of real value to the team.

Agile Test team can’t set up automated testing tools without team members who can write code.  Automating tests is software development. It must be done with the same care and thought that goes into writing production code.This poses a significant hurdle for testers who lack coding skills. That challenge is best met by mastering those skills, not by shying away from projects that demand them

Better software is the result of running the right tests and continually re-evaluating which tests are the right ones,

Automated unit tests check the behavior of individual functions/methods and object interactions. They’re to run often, and provide feedback in minutes. Automated acceptance tests usually check the behavior of the system end-to-end. (Although,  sometimes they bypass the GUI, checking the underlying business logic.) They’re  typically run on checked in code on an ongoing basis, providing feedback in an hour  or so. Agile projects favor automated tests because of the rapid feedback they provide.

Agile Testing Quadrant

TheAgileTestingQuadrants.png

In the agile testing quadrant depicted above, The order in which quadrants numbered  has no relationship to when the different types of testing are done. Most projects would start with Q2 tests, because those are where you get the examples that turn into specifications and tests that drive coding, along with prototypes and the like. The quadrants are a taxonomy to help teams plan their testing and make sure they have all the resources they need to accomplish it.The timing of the various types of tests depends on the risks of each project, the customers’ goals for the product, whether the team is working with legacy code or on a greenfield project, and when resources are available to do the testing.

The lower left quadrant represents test-driven development, which is a core agile development practice. The quadrants on the left include tests that support the team as it develops the product.The testing done in Quadrants 1 and 2 are more requirements specification and design aids than what we typically think of as testing.

Unit tests verify functionality of a small subset of the system, such as an object or method. Component tests verify the behavior of a larger part of the system, such as a group of classes that provide some service [Meszaros, 2007]. Both types of tests are usually automated with a member of the xUnit family of test automation tools. We refer to these tests as programmer tests, developerfacing tests, or technology-facing tests. They enable the programmers to measure what Kent Beck has called the internal quality of their code [Beck, 1999].
A major purpose of Quadrant 1 tests is test-driven development (TDD) or test-driven design. The process of writing tests first helps programmers design their code well. These tests let the programmers confidently write code to deliver a story’s features without worrying about making unintended changes to the system. They can verify that their design and architecture decisions are appropriate. Unit and component tests are automated and written in the same programming language as the application. A business expert probably couldn’t understand them by reading them directly, but these tests aren’t intended for customer use. In fact, internal quality isn’t negotiated with the customer; it’s defined by the programmers. Programmer tests are normally part of an automated process that runs with every code check-in, giving the team instant, continual feedback about their internal quality

The tests in Quadrant 2 also support the work of the development team, but at a higher level. These business-facing tests, also called customer-facing tests and customer tests, define external quality and the features that the customers want.

They describe the details of each story. Business-facing tests run at a functional level, each one verifying a business satisfaction condition. They’re written in a way business experts can easily understand using the business domain language. In fact, the business experts use these tests to define the external quality of the product and usually help to write them.

One of the most important purposes of tests in these two quadrants (Q1 and Q2) is to provide information quickly and enable fast troubleshooting. They must be run frequently in order to give the team early feedback in case any behavior changes unexpectedly. All of these tests should be run as part of an automated continuous integration, build, and test process.

Technology-facing tests in Quadrant 4 are intended to critique product characteristics such as performance, robustness, and security. The types of tests that fall into the fourth quadrant are just as critical to agile development as to any type of software development. These tests are technology-facing. Creating and running these tests might require the use of specialized tools and additional expertise.

Quadrant 4 Automation tools

Native database tools

  •  SQL, data import tools
  • Shell scripting 
  • Monitoring tools examples
    • jConsole
      • Application bottlenecks, memory leaks
    • jProfiler
      • Database and bean usage
  • Commercial load test tools
    • Loadrunner
    • Silk Performer
  • Open source test tools
    • jMeter
    • The Grinder
    • jUnitPerf
  • Performance test providers
  • Multiple sites

The short iterations of agile development give your team a chance to learn and experiment with the different testing quadrants

Test Automation Backlog

Maintain a test automation backlog for the project that contains all needed automation tasks and identified improvements. Then target a few items from the backlog every sprint, in no time team will start to see the new regression test suite taking shape. Occasionally, stories from the test automation backlog may require dedicated developer time to implement and consequently some buy-in from the product owner in order to proceed. However, it should not be difficult to convince the product owner about the value of such stories if everyone on team is committed to quality. A test automation backlog could contain a prioritized list of items such as:

  • Parameterize the test environment for test execution.
  • Integrate with Continuous Integration.
  • Enhance reporting mechanism.
  • Provide an option to attach error logs in notification emails.
  • Collect performance metrics for workflow scenarios.
  • Add tests to check for concurrent execution of critical test cases.

Agile test automation pyramid

TA-pyramid 2 Figure 1

In the traditional Test Automation Pyramid view, most if not all of the effort was in developing UI-centric functional tests that explored the application via the GUI. There might be some lower-level tests and a few unit tests, but teams mostly staid at the upper tier.

The agile test automation pyramid is a strategy that attempts to alter the Test automation pyramid and provide the Agile Test automation pyramid depicted in the figure above.

Agile testing relies more on automation. It requires a much greater contribution from developers. And it has a different basic philosophy – to prevent bugs.

The first change is taking a whole-team view. Instead of the testers being responsible for testing AND writing all of the test automation, it becomes a whole-team responsibility. The developers take most of the ownership for unit-level automation, but testers can operate here as well.

At the base of the test automation pyramid is unit testing. Unit testing should be the foundation of a solid test automation strategy and as such represents the largest part of the pyramid.(50 % to 60 % – Unit Tests)

The upper tier focuses on limited GUI-based automation tests. Usually, these are longer running, core customer usage workflows that are best implemented at this level.

Automated user interface testing (0% to 10%) is placed at the top of the test automation pyramid because we want to do as little of it as possible.

The testers typically operate on these, but there are very few tests within this tier. And remember that the developers can operate here as well. The two layers are met by middle-tier automation. This is often the domain of the open source ATDD/BDD tools, such as: FitNesse, Cucumber, JBehave, Robot Framework, and others. The middle tier is the Acceptance Test tier / API Layer tests are performed.

One key thing to note is that traditional automation was often a one-tool operation, with Mercury/HP commercial tooling (Quick Test Professional or QTP) leading that space.  The agile approach is tool agnostic, but also aggregates tools that are appropriate for each layer. Therefor no “one size fits all” thinking is allowed. For example, these are common used tools at each tier:

  1. UI tier:  Selenium, Watir, or traditional QTP
  2. Middle tier:  FitNesse, Robot Framework, and Cucumber
  3. Unit tier:  xUnit family variants for example JUnit or NUnit for Java and .Net respectively

The other consideration is that there are extensions to many of these. For example, both Robot Framework and Cucumber have Selenium plug-ins so that they can ‘drive’ the UI as well as the middle tier. This implies that the middle tier tooling, and automated tests for that matter, can extend or blend into the lower and upper tiers.

Rather than investing in extensive, heavyweight step-by-step manual test scripts in  Word or a test management tool, we capture expectations in a format supported by  automated test frameworks like FIT/Fitnesse. The test could be executed manually,  but more importantly that same test artifact becomes an automated test when the  programmers write a fixture to connect the test to the software under test.

Test-driven development (TDD)

One key benefit of agile software development is the ability to make changes to the code quickly and easily. As the team embraces changing requirements as a part of a living, breathing system, they need to know that their changes will not cause a domino effect of broken code. Ideally, the design of the system will be so simple and well isolated that any possible effects of a change will be readily apparent. In the real world, this is not always the case, especially when working with legacy systems. In this case, something must be implemented to ensure that if a change affects other areas of the system, there will be an immediate red-flag, rather than having to chase down the issues later.

After experimenting with different ways to enable this flexibility, agile teams hit upon the practice of Test Driven Development, which involves producing automated unit tests for production code before you write that production code. Instead of writing tests afterward (or, more typically, never writing those tests), you always begin with a unit test. For every small chunk of functionality in production code, you first build and run a small, focused test that specifies and validates what the code will do. This test might not even compile, at first, because all of the classes and methods it requires may not yet exist. Nevertheless, it functions as an executable specification. You then get it to compile with minimal production code, so that you can run it and watch it fail. You then produce exactly as much code as will enable that test to pass. Sometimes you expect it to fail, and it passes, which is also useful information.

This technique feels odd, at first, to quite a few programmers who try it. It’s a bit like rock climbers inching up a rock wall, placing anchors in the wall as they go. Why go to all this trouble? Surely it slows you down considerably. The answer is that it only makes sense if you end up relying heavily and repeatedly on those anchors (unit tests) later. Those who practice Test Driven Development regularly claim that the benefits of those unit tests more than pay back the effort required to write them.

For Test-First work, you will typically use one of the xUnit family of automated unit test frameworks (JUnit for Java, NUnit for C#, etc). These frameworks make it quite straightforward to create, run, organize, and manage large suites of unit tests. Test Driven Development has grown to be such a useful tool that it is well integrated into most of the major IDEs, including Visual Studio and Eclipse.

Test-driven development (TDD) is an advanced technique of using automated unit tests to drive the design of software and force decoupling of dependencies. The result of using this practice is a comprehensive suite of unit tests that can be run at any time to provide feedback that the software is still working. This technique is heavily emphasized by those using Agile development methodologies

The motto of test-driven development is “Red, Green, Refactor.”

  • Red: Create a test and make it fail.
  • Green: Make the test pass by any means necessary.
  • Refactor: Change the code to remove duplication in your project and to improve the design while ensuring that all tests still pass.

The Red/Green/Refactor cycle is repeated very quickly for each new unit of code.

Follow these steps (slight variations exist among TDD practitioners):

  1. Understand the requirements of the story, work item, or feature that you are working on.
  2. Red: Create a test and make it fail.
    1. Imagine how the new code should be called and write the test as if the code already existed. You will not get IntelliSense because the new method does not yet exist.
    2. Create the new production code stub. Write just enough code so that it compiles.
    3. Run the test. It should fail. This is a calibration measure to ensure that your test is calling the correct code and that the code is not working by accident. This is a meaningful failure, and you expect it to fail.
  3. Green: Make the test pass by any means necessary.
    1. Write the production code to make the test pass. Keep it simple.
    2. Some advocate the hard-coding of the expected return value first to verify that the test correctly detects success. This varies from practitioner to practitioner.
    3. If you’ve written the code so that the test passes as intended, you are finished. You do not have to write more code speculatively. The test is the objective definition of “done.” The phrase “You Ain’t Gonna Need It” (YAGNI) is often used to veto unnecessary work. If new functionality is still needed, then another test is needed. Make this one test pass and continue.
    4. When the test passes, you might want to run all tests up to this point to build confidence that everything else is still working.
  4. Refactor: Change the code to remove duplication in your project and to improve the design while ensuring that all tests still pass.
    1. Remove duplication caused by the addition of the new functionality.
    2. Make design changes to improve the overall solution.
    3. After each refactoring, rerun all the tests to ensure that they all still pass.
  5. Repeat the cycle. Each cycle should be very short, and a typical hour should contain many Red/Green/Refactor cycles.

Characteristics of a Good Unit Test

A good unit test has the following characteristics.

  • Runs fast, runs fast, runs fast. If the tests are slow, they will not be run often.
  • Separates or simulates environmental dependencies such as databases, file systems, networks, queues, and so on. Tests that exercise these will not run fast, and a failure does not give meaningful feedback about what the problem actually is.
  • Is very limited in scope. If the test fails, it’s obvious where to look for the problem. Use few Assert calls so that the offending code is obvious. It’s important to only test one thing in a single test.
  • Runs and passes in isolation. If the tests require special environmental setup or fail unexpectedly, then they are not good unit tests. Change them for simplicity and reliability. Tests should run and pass on any machine. The “works on my box” excuse doesn’t work.
  • Often uses stubs and mock objects. If the code being tested typically calls out to a database or file system, these dependencies must be simulated, or mocked. These dependencies will ordinarily be abstracted away by using interfaces.
  • Clearly reveals its intention. Another developer can look at the test and understand what is expected of the production code.

Rich feature sets provided by the automation framework would essentially reduce the effort put in by the development teams in the TDD workflow. Here are some of the aspects of automation frameworks that can pay rich dividends: Ease of tracking: Unit tests would be stored in a central repository (part of the automation framework) with all the development team members submitting their unit tests to it. Tests would be stored in hierarchical folder structure based on the product features and its components. With this, viewing and tracking of unit tests within and across teams would be smoother. Traceability of unit tests: Automation frameworks can ensure that each product requirement has a unit test associated with it. This ensures that all requirements are developed as part of the TDD process, thus avoiding development slippages.

Improving the development and review process: Automation infrastructure can facilitate tracking of all requirements by associating them with a developer and reviewer(s). This would ensure that development and review processes are organized. Unit test execution: A good automation framework ensures quick running of automated unit tests. The tests could be executed selectively for a component, set of features or the product itself. Reporting of test execution results: Results of the automated unit test for a component/ feature would be sent to the respective developer; this ensures quick reporting and cuts short the response time in refactoring unit tests from the developer. Automation infrastructure components: Automation frameworks could facilitate:

  • Cross-platform testing
  • Compatibility testing

xUnit frameworks

Developers may use computer-assisted testing frameworks, such as xUnit, to create and automatically run the test cases. Xunit frameworks provide assertion-style test validation capabilities and result reporting. These capabilities are critical for automation as they move the burden of execution validation from an independent post-processing activity to one that is included in the test execution. The execution framework provided by these test frameworks allows for the automatic execution of all system test cases or various subsets along with other features.

Fakes, mocks and integration tests

Unit tests are so named because they each test one unit of code. A complex module may have a thousand unit tests and a simple module may have only ten. The tests used for TDD should never cross process boundaries in a program, let alone network connections. Doing so introduces delays that make tests run slowly and discourage developers from running the whole suite. Introducing dependencies on external modules or data also turns unit tests into integration tests. If one module misbehaves in a chain of interrelated modules, it is not so immediately clear where to look for the cause of the failure.

When code under development relies on a database, a web service, or any other external process or service, enforcing a unit-testable separation is also an opportunity and a driving force to design more modular, more testable and more reusable code. Two steps are necessary:

  1. Whenever external access is needed in the final design, an interface should be defined that describes the access available.
  2. The interface should be implemented in two ways, one of which really accesses the external process, and the other of which is a fake or mock. Fake objects need do little more than add a message such as “Person object saved” to a trace log, against which a test assertion can be run to verify correct behaviour. Mock objects differ in that they themselves contain test assertions that can make the test fail, for example, if the person’s name and other data are not as expected.

Fake and mock object methods that return data, ostensibly from a data store or user, can help the test process by always returning the same, realistic data that tests can rely upon. They can also be set into predefined fault modes so that error-handling routines can be developed and reliably tested. In a fault mode, a method may return an invalid, incomplete or null response, or may throw an exception. Fake services other than data stores may also be useful in TDD: A fake encryption service may not, in fact, encrypt the data passed; a fake random number service may always return 1. Fake or mock implementations are examples of dependency injection.

A Test Double is a test-specific capability that substitutes for a system capability, typically a class or function, that the UUT depends on. There are two times at which test doubles can be introduced into a system: link and execution. Link time substitution is when the test double is compiled into the load module, which is executed to validate testing. This approach is typically used when running in an environment other than the target environment that requires doubles for the hardware level code for compilation. The alternative to linker substitution is run-time substitution in which the real functionality is replaced during the execution of a test cases. This substitution is typically done through the reassignment of known function pointers or object replacement.

Test doubles are of a number of different types and varying complexities:

  • Dummy – A dummy is the simplest form of a test double. It facilitates linker time substitution by providing a default return value where required.
  • Stub – A stub adds simplistic logic to a dummy, providing different outputs.
  • Spy – A spy captures and makes available parameter and state information, publishing accessors to test code for private information allowing for more advanced state validation.
  • Mock – A mock is specified by an individual test case to validate test-specific behavior, checking parameter values and call sequencing.
  • Simulator – A simulator is a comprehensive component providing a higher-fidelity approximation of the target capability (the thing being doubled). A simulator typically requires significant additional development effort.

A corollary of such dependency injection is that the actual database or other external-access code is never tested by the TDD process itself. To avoid errors that may arise from this, other tests are needed that instantiate the test-driven code with the “real” implementations of the interfaces discussed above. These are integration tests and are quite separate from the TDD unit tests. There are fewer of them, and they must be run less often than the unit tests. They can nonetheless be implemented using the same testing framework, such as xUnit.

Integration tests that alter any persistent store or database should always be designed carefully with consideration of the initial and final state of the files or database, even if any test fails. This is often achieved using some combination of the following techniques:

  • The TearDown method, which is integral to many test frameworks.
  • try…catch…finally exception handling structures where available.
  • Database transactions where a transaction atomically includes perhaps a write, a read and a matching delete operation.
  • Taking a “snapshot” of the database before running any tests and rolling back to the snapshot after each test run. This may be automated using a framework such as Ant or NAnt or a continuous integration system such as CruiseControl.
  • Initialising the database to a clean state before tests, rather than cleaning up after them. This may be relevant where cleaning up may make it difficult to diagnose test failures by deleting the final state of the database before detailed diagnosis can be performed.

Continuous Integration

CI is a software development practice, where Team members integrate their work frequently, usually with each check-in to the version control system . An automated server builds the system, runs all the unit tests and reports “green” or “red” to indicate if the build is stable/broken.  Building and Testing are done on a dedicated machine to eliminate “it works on my machine” dependencies.

Over the past three years, Ancestry.com, the world’s largest online family history resource, underwent a significant transformation in technology and infrastructure. Starting with the adoption of Agile development practices, the company evolved to a Continuous Delivery model that enables code release whenever the business requires it. Transitioning from large, weekly or bi-weekly software rollouts to smaller, incremental updates has allowed Ancestry.com to increase responsiveness and deliver new features to customers more quickly.

Continuous Integration features

  • Very High ROI. One of the first engineering practices an Agile teams should adopt.
  • Many open source and commercial tools available
  • Keep builds and tests fast (move slow running tests to a nightly build)
  • Adopt a “Stop the Line” mentality (all work stops until a broken build is fixed)
  • Test in a clone of production (Every environmental difference results in risk)

Continuous integration (CI) is the practice, in software engineering, of merging all developer working copies with a shared mainline several times a day. It was first named and proposed as part of extreme programming (XP). Its main aim is to prevent integration problems, referred to as “integration hell” in early descriptions of XP. CI can be seen as an intensification of practices of periodic integration advocated by earlier published methods of incremental and iterative software development, such as the Booch method. CI isn’t universally accepted as an improvement over frequent integration, so it is important to distinguish between the two as there is disagreement about the virtues of each.

CI was originally intended to be used in combination with automated unit tests written through the practices of test-driven development. Initially this was conceived of as running all unit tests and verifying they all passed before committing to the mainline. This helps avoid one developer’s work in progress breaking another developer’s copy. If necessary, partially complete features can be disabled before committing using feature toggles.

Later elaborations of the concept introduced build servers, which automatically run the unit tests periodically or even after every commit and report the results to the developers. The use of build servers (not necessarily running unit tests) had already been practised by some teams outside the XP community. Nowadays, many organisations have adopted CI without adopting all of XP.

In addition to automated unit tests, organisations using CI typically use a build server to implement continuous processes of applying quality control in general — small pieces of effort, applied frequently. In addition to running the unit and integration tests, such processes run additional static and dynamic tests, measure and profile performance, extract and format documentation from the source code and facilitate manual QA processes. This continuous application of quality control aims to improve the quality of software, and to reduce the time taken to deliver it, by replacing the traditional practice of applying quality control after completing all development. This is very similar to the original idea of integrating more frequently to make integration easier, only applied to QA processes.

In the same vein the practice of continuous delivery further extends CI by making sure the software checked in on the mainline is always in a state that can be deployed to users and makes the actual deployment process very rapid.

Twist at AutoTrader

Testing was growing unwieldy for AutoTrader.co.uk, the UK’s #1 automotive website, with multiple tools, lack of  automation, and increasing test maintenance. Enter Twist—the test automation platform that reduced testing effort  by 95%, achieved 89% test coverage, and minimized testing time by 75%. Furthermore, Twist removed the biggest  pain-point, by drastically reducing the overhead of test maintenance. With Twist, the team could add real value and  test new functionality at a rapid pace

Sustaining quality for a dynamic and complex application can end up in a vicious cycle of modifying tests, testing
new functionality, and optimizing existing tests. AutoTrader.co.uk is the UK’s most popular motoring website, and
the fourth busiest search engine, getting over 10 million unique visitors every month. With such high traffic
volumes, optimal test coverage is not just nice-to-have, but a critical necessity. However, testing such a complex,
and ever changing application was a maintenance-heavy, lengthy, and expensive exercise that was further
complicated by difficult testing tools. Auto Trader liked Twist’s support for writing test specifications in plain English,
automating robust regression tests, and integrating with continuous integration (CI) and source control products,
and decided to give it a try. Since 2008, Twist has dramatically reduced AutoTrader.co.uk’s testing effort and
increased test coverage. It has also improved collaboration between business analysts (BAs) and quality analysts
(QAs) on tests, by simplifying technically complex tests to readable English constructs. Twist’s highly refactored,
modular test suite now exhaustively tests the site in 12 minutes, and can easily keep pace with application changes.
Additionally, Twist’s usability makes it highly popular across the teams.

PEARL XII : Kanban : Application of Lean software development to Agile Methodologies

PEARL XII : Kanban : Application of Lean software development to Agile Methodologies.

Most agile m­ethods such as Scrum and XP are already well aligned with lean principles. In 2004, David Anderson pioneered a more direct implementation of Lean Thinking and Theory of Constraints to software development. Under the guidance of experts such as Don Reinertsen, this evolved into what David called a “Kanban system for software development”, and which most people now simply refer to as “Kanban”. ­So while Kanban as applied to software development is new, Kanban as used in Lean Production is over a half century old.

A software development lifecycle process or a project management process could be said to be “lean” if it was observed to be aligned with the values of the Lean Software Development movement and the principles of Lean Software Development.

Bob Charette was invited but unable to attend the 2001 meeting at Snowbird, Utah, where the Manifesto for Agile Software Development was authored. Despite missing this historic meeting, Lean Software Development was considered as one of several Agile approaches to software development. Jim Highsmith dedicated a chapter of his 2002 book to an interview with Bob about the topic. Later, Mary & Tom Poppendieck went on to author a series of 3 books. During the first few years of the 21st Century, Lean principles were used to explain why Agile methods were better. Lean explained that Agile methods contained little “waste” and hence produced a better economic outcome. Lean principles were used as a “permission giver” to adopt Agile methods.

Lean Software Development is an iterative agile methodology originally developed by Mary and Tom Poppendieck. Lean Software Development owes much of its principles and practices to the Lean Enterprise movement, and the practices of companies like Toyota. Lean Software Development focuses the team on delivering Value to the customer, and on the efficiency of the “Value Stream,” the mechanisms that deliver that Value. The main principles of Lean include:

  1. Eliminating Waste
  2. Amplifying Learning
  3. Deciding as Late as Possible
  4. Delivering as Fast as Possible
  5. Empowering the Team
  6. Building Integrity In
  7. Seeing the Whole

Lean eliminates waste through such practices as selecting only the truly valuable features for a system, prioritizing those selected, and delivering them in small batches. It emphasizes the speed and efficiency of development workflow, and relies on rapid and reliable feedback between programmers and customers. Lean uses the idea of work product being “pulled” via customer request. It focuses decision-making authority and ability on individuals and small teams, since research shows this to be faster and more efficient than hierarchical flow of control. Lean also concentrates on the efficiency of the use of team resources, trying to ensure that everyone is productive as much of the time as possible. It concentrates on concurrent work and the fewest possible intra-team workflow dependencies. Lean also strongly recommends that automated unit tests be written at the same time the code is written.

Perhaps the most important Lean activity is to build a value stream map. This means to break down a process into individual steps, and identify which steps add value and which steps do not, thus adding to the waste (muda). The goal, then, is to eliminate the waste and improve the value-added steps (kaizen). An important lean tool helping to manage the work flow is the concept of pull system, which is usually visualized using a Kanban board. In general, we can define the Kanban software process  as a WIP (Work In Process) limited pull system visualized by the Kanban board.

In recent years, Lean Software Development has really emerged as its own discipline related to, but not specifically a subset of the Agile movement. This evolution started with the synthesis of ideas from Lean Product Development and the work of Donald G. Reinertsen and ideas emerging from the non-Agile world of large scale system engineering and the writing of James Sutton and Peter Middleton.  David also synthesized the work of Eli Goldratt and W. Edwards Deming and developed a focus on flow rather than waste reduction . At the behest of Reinertsen around 2005, David introduced the use of kanban systems that limit work-in-progress and “pull” new work only when the system is ready to process it. Alan Shalloway added his thoughts on Lean software development in his 2009 book on the topic. Since 2007, the emergence of Lean as a new force in the progress of the software development profession has been focused on improving flow, managing risk, and improving (management) decision making. Kanban has become a major enabler for Lean initiatives in IT-related work. It appears that a focus on flow, rather than a focus on waste elimination, is proving a better catalyst for continuous improvement within knowledge work activities such as software development.

The Lean Systems Society published its credo at the 2012 Lean Software & Systems Conference. This was based on a set of values published a year earlier. Those values include:

  • Accept the human condition
  • Accept that complexity & uncertainty are natural to knowledge work
  • Work towards a better Economic Outcome
  • While enabling a better Sociological Outcome
  • Seek, embrace & question ideas from a wide range of disciplines
  • A values-based community enhances the speed & depth of positive change

Kanban is an agile methodology for managing the creation of products with an emphasis on continual delivery while not overburdening the development team. Like Scrum, Kanban is a process designed to help teams work together more effectively.

Kanban is a method for managing knowledge work with an emphasis on just-in-time delivery while not overloading the team members. In this approach, the process, from definition of a task to its delivery to the customer, is displayed for participants to see and team members pull work from a queue.

Kanban in the context of software development can mean a visual process management system that tells what to produce, when to produce it, and how much to produce inspired by the Toyota Production System and Lean manufacturing.

The name ‘Kanban’ originates from Japanese and translates roughly as “signboard” or “billboard”. The Kanban method in one of the Kanban methods available today. It was formulated by David J. Anderson is an approach to incremental, evolutionary process and systems change for organizations. It uses a work-in-progress limited pull system as the core mechanism to expose system operation (or process) problems and stimulate collaboration to continuously improve the system. One example of such a pull system is a kanban system, and it is after this popular form of a work-in-progress, limited pull system that the method is named.

In Lean Software Development, the kanban are virtual and often tracked by setting a maximum number for a given step in the workflow of a work item type. In some implementations, electronic systems keep track of the virtual kanban and provide a signal when new work can be started. The signal can be visual or in the form of an alert such as an email.

Virtual kanban systems are often combined with visual controls to provide a visual virtual kanban system representing the workflow of one or several work item types. Such systems are often referred to as “kanban boards” or “electronic kanban systems

Visual control systems are valuable in changing behavior because they display status in an easy-to-see format. This ensures that everyone has a shared understanding of work status and process constraints. Transparency is a key to achieving organizational change

The Kanban method is rooted in four basic principles:

Start with what you do now

The Kanban method does not prescribe a specific set of roles or process steps. The Kanban method starts with the roles and processes you have and stimulates continuous, incremental and evolutionary changes to your system. The Kanban method is a change management method.

Agree to pursue incremental, evolutionary change

The organization (or team) must agree that continuous, incremental and evolutionary change is the way to make system improvements and make them stick. Sweeping changes may seem more effective but have a higher failure rate due to resistance and fear in the organization. The Kanban method encourages continuous small incremental and evolutionary changes to your current system.

Respect the current process, roles, responsibilities and titles

It is likely that the organization currently has some elements that work acceptably and are worth preserving. We must also seek to drive out fear in order to facilitate future change. By agreeing to respect current roles, responsibilities and job titles we eliminate initial fears. This should enable us to gain broader support for our Kanban initiative. Perhaps presenting Kanban against an alternative more sweeping approach that would lead to changes in titles, roles, responsibilities and perhaps the wholesale removal of certain positions will help individuals to realize the benefits.

Leadership at all levels

Acts of leadership at all levels in the organization from individual contributors to senior management should be encouraged.

Kanban Vs Agile

Kanban systems are an approach to scheduling  work. Kanban shares with typical AMs the fact that  requirements are expressed in atomic features (also  known as user stories, work items, Minimum  Marketable Features, or MMF), to be implemented  incrementally. To this purpose, AMs use time-boxed  iterations. – at the beginning of the iteration, the team meets and chooses features from their backlog  that can be done by the end of the iteration. Over the  course of the iteration, they develop those stories,  and at the end of the iteration, they ship them .
Kanban systems are different because they focus on a continuous flow of work, and disregard fixed  iterations. When needed, the team chooses a subset  of features from the backlog and moves them to the  Kanban board. Then, it develops these features one
after the other. Work is delivered as soon as it’s  ready, and the team only works on one – or very few – feature at a time.

Six core practices of Kanban

Anderson identified five core properties that had been observed in each successful implementation of the Kanban method. They were later relabeled as practices and extended with the addition of a sixth.

  1. Visualize
    The workflow of knowledge work is inherently invisible. Visualising the flow of work and making it visible is core to understanding how work proceeds. Without understanding the workflow, making the right changes is harder.
    A common way to visualise the workflow is to use a card wall with cards and columns. The columns on the card wall representing the different states or steps in the workflow.
  2. Limit WIP
    Limiting work-in-process implies that a pull system is implemented on parts or all of the workflow. The pull system will act as one of the main stimuli for continuous, incremental and evolutionary changes to your system.
    The pull system can be implemented as a kanban system, a CONWIP system, a DBR system, or some other variant. The critical elements are that work-in-process at each state in the workflow is limited and that new work is “pulled” into the new information discovery activity when there is available capacity within the local WIP limit.
    WIP limits encourage everyone to work as a team and prevent any one individual from getting too far ahead of anyone else.
  3. Manage flow
    The flow of work through each state in the workflow should be monitored, measured and reported. By actively managing the flow the continuous, incremental and evolutionary changes to the system can be evaluated to have positive or negative effects on the system. Kanban elevates awareness of constraints and forces the team to address them before they can bring additional work into the queue. New work is pulled into the system when there is capacity to handle it, rather than being pushed into the system based on demand. With Kanban, the focus of management and the team becomes the flow of work, not the utilization of the team.
    While Scrum has retrospectives at the end of each iteration to address these items, Kanban explicitly points out constraints in real time and encourages the team to address them as they arise.
  4. Make policies explicit
    Until the mechanism of a process is made explicit, it is often hard or impossible to hold a discussion about improving it. Without an explicit understanding of how things work and how work is actually done, any discussion of problems tends to be emotional, anecdotal and subjective. With an explicit understanding it is possible to move to a more rational, empirical, objective discussion of issues. This is more likely to facilitate consensus around improvement suggestions.
  5. Implement feedback loops
    Collaboration to review flow of work and demand versus capability measures, metrics and indicators coupled with anecdotal narrative explaining notable events is vital to enabling evolutionary change. Organizations that have not implemented the second level of feedback – the operations review – have generally not seen process improvements beyond a localized team level. As a result, they have not realized the full benefits of Kanban observed elsewhere.
  6. Improve collaboratively, evolve experimentally (using models and the scientific method)
    The Kanban method encourages small continuous, incremental and evolutionary changes that stick. When teams have a shared understanding of theories about work, workflow, process, and risk, they are more likely to be able to build a shared comprehension of a problem and suggest improvement actions which can be agreed by consensus.
    The Kanban method suggests that a scientific approach is used to implement continuous, incremental and evolutionary changes.

Common models used are:

  • Theory of constraints (the study of bottlenecks)
  • Deming System of Profound Knowledge (a study of variation and how it affects processes)
  • Lean economic model (based on the concept of elimination of “waste” (or muda, muri and mura)).

Open Kanban

An open source, Agile and Lean based method to deliver value for knowledge work like Information Technology, Software Development, Business, Product Development or Personal organization. On the Lean side it is inspired on the work of Taiichi Ohno (Toyota Production System), Eliyahu Goldratt (Theory of Constraints) and Edward Deming. On the Agile side it takes inspiration from the Agile manifesto signers, and in addition contributions from Alan Shalloway’s Kanban for Teams, Corey Ladas Scrumban and David Anderson’s early Kanban work.

It innovates by making the whole method fully open source and free to improve or modify. Open Kanban was written by Joseph Hurtado, and it has been translated by members of the community to French, Italian, Russian and Ukrainian.

Steps to Get Started

  • Agree on a set of goals for introducing Kanban.
  • Map the value stream.
  • Define some point where you want to control  input. Define what is upstream and who are the upstream stakeholders.
  • Define some exit point beyond which you do  not intend to control.
  • Define a set of work items types based on the work requests that come from the upstream stakeholders.
  • Analyze the demand for each work item type.
  • Meet with the upstream and downstream stakeholders.
  • Create a board/card wall to track the value stream you are controlling.
  • Optionally, create an electronic system to track and report the same.
  • Agree with the team to have a standup meeting in front of the board; invite upstream and downstream stakeholders .
  • Agree to have regular operations review meeting for retrospective analysis of the  process; invite upstream and downstream stakeholders .
  • Train the team on the new board, WIP limits and the pull system. Nothing else in  their world should have changed.

VersionOne Kanban Boards

VersionOne provides a single platform to manage all your projects, from Scrum to Kanban and everything in between, without sacrificing cross-project visibility. Visualize, manage and optimize your workflow using VersionOne’s custom Kanban boards.

With some teams practicing Scrum and others choosing Kanban, it can be challenging for managers to get a clear, integrated view of status across all projects. VersionOne’s Kanban boards make it easy to manage all projects in a single tool without sacrificing cross-project visibility and consolidated reporting. When teams are practicing lean methods within iterations, VersionOne provides the Kanban capabilities to help agile teams visualize and manage their workflow processes.

Agile teams can Create custom Kanban boards using named columns to illustrate each stage of  workflow process, and an expedited track for high priority items. Work items are visually represented using story cards and can be pulled from prioritized queue and moved through the workflow easily using drag-and-drop software functionality.

Managing the Kanban board in VersionOne provides real-time visibility for the entire team with full accessibility for remote members. Agile Teams see at a glance what work items are in progress and where they are in the development process. Visual cues within the Kanban board identify items that are blocked, held up in wait queue, or have exceeded their aging threshold, providing an early indicator of potential problems.

VersionOne software helps maximize  flow of work by setting explicit WIP limits across single or multiple contiguous columns to determine how many items can be worked on at each stage of  process. Visual cues within the Kanban board lets team know when WIP limits are exceeded to trigger conversations and refocus the agile team’s efforts on the right work to ensure a continuous flow.

VersionOne enables to Track and measure the average time to complete a work item using Cycle Cycle Time reporting capabilities. The ability to track cycle time metrics gives visibility into past performance to help  increase the predictability of  delivery and drive continuous process improvement.

The Cycle Time clock starts when work begins on a request and ends when the item is ready for delivery. Cycle time is a mechanical measure of process capability.

VersionOne can be used Identify and resolve bottlenecks using Cumulative Flow reports to see how well items are flowing through the process, Determine where work is stalled and where you should focus improvement efforts.

The Cumulative Flow diagram provides a graphic depiction of how work items are moving through various statuses on the way to being “Done”. It shows the total scope of a project, grouped by status, and thus lets us know how much of that scope is in a particular status at a given time.

As highlightedd, the use of a basic Kanban board (manual and/or virtual for distributed teams)  serves to bring everyone onto the same page, improve focus and prompt questions that lead to revision of policies and practices. When the team’s  experience grows, the Kanban board will change  to expand and refine the workflow representation, thereby reflecting a more sophisticated level