PEARL XV : Beyond Scrum : A Scalable Agile Framework with Continuous Integration

PEARL XV : Beyond Scrum :  A Scalable Agile Framework with Continuous Integration using Test Automation and Build for Large Distributed Agile Projects.

scalable agile process

Scrum is the most popular Agile technique, but it doesn’t scale well. And while Scrum improves the effectiveness of individual teams, productivity gains fall off sharply on large projects with many teams.

Yet Agile methods have been used in very large commercial and open source projects to increase productivity, quality and release frequency. Here are a couple of examples:

  • Facebook incorporates code from 600 developers while delivering two releases per day.
  • The Google Android project utilizes thousands of contributors spread across the world.
  • Flickr reached a level of 10 deployments per day.

Can we learn from these companies and projects? What types of techniques and tools did they use to achieve those results?

This section will cover the problems with Scrum, changes in approach to help scalability and methods for supporting distributed teams and continuous delivery.

Problems With Scrum
Scrum techniques have been very successful in improving the effectiveness of individual development teams, employing concepts like self-directed co-located teams, time-boxed sprints, and regular customer feedback from working software. Yet many organizations have run into obstacles when trying to apply Scrum techniques to large projects. For example, no effective techniques have evolved to coordinate the work of multiple Scrum teams and manage dependencies among them. The “Scrum of Scrums” approach of holding meetings with representatives of every team becomes increasingly time-consuming and unwieldy as the number of teams multiply.

In part, this is because some of the assumptions underlying Scrum are too restrictive for large organizations, or clash with business requirements. Many groups refuse to be limited to co-located teams because they are already distributed, they want to take advantage of the global market for development talent, or simply because many of their employees work from home several days a week.

Many groups need to share key personnel such as architects, UI designers, and database specialists across many projects, and cannot assign them to a single team. Other companies need to fix bugs and release new functionality more frequently than the 2-8 week cycles typical of Scrum teams. This is particularly true of those providing web- and cloud-based applications, where customers expect a constant flow of enhancements.

This is not to say that Scrum practices have no place in a large development environment. But it is now clear that many organizations need to go “Beyond Scrum” and find new practices to manage distributed contributors and large, complex projects.

Changes in Approach That Help Scalability
The large commercial and open source projects that have successfully scaled Agile typically depart from conventional Scrum practices in several areas.

No Scrum meetings: Sprint planning meetings, retrospectives and Scrum-of-Scrum meetings are time-consuming, usually require that everyone be in one room, and usually don’t do a very good job of coordinating across teams. That’s why large-scale projects find ways to use online collaboration and planning tools to coordinate work within and across teams, with fewer meetings and conference calls.

“Pull,” “continuous flow,” and “publish what’s ready”: Although Scrum practices are far more agile than the waterfall methods they replaced, they still impose a degree of inflexibility. Once a sprint plan is complete, the features to be delivered are fixed, and so is the time frame to deliver them (usually 2-8 weeks).

Scalable projects typically use pull and continuous flow techniques (especially Kanban), so developers are always working on the highest priority tasks, and new functionality can be released as soon as it is ready.

Code review workflows (long used by open source projects) can be used to select contributions from hundreds of contributors and publish what’s ready. By helping organizations scale to more contributors and release more frequently, code review workflows can become a key building block of Scalable Agile.

Diverse contributors: Classic Scrum practices are designed for co-located teams of 8-10 members. But, in reality, large projects need to incorporate work from individual contributors, shared resources (e.g., architects and DBAs), outsourcing companies, and business partners as well as teams. Collaboration tools and code review workflows are central to meshing the work of these diverse contributors.

A Scalable Agile Process Framework
The question, then, is how can we apply the new approaches as a coherent whole?

Instead of each Scrum team having its own backlog, product management (or product owners) maintain a single project-wide backlog, with tasks sorted in priority order.

At any time, contributors can pull tasks from the top of the backlog into their own “Current Work” list. Ideally they will pull the highest-priority task, but if they do not have the necessary skills or resources they can move down the stack to find another high-priority assignment.

This process ensures that the highest-priority tasks are addressed first. If an urgent bug fix or a key customer feature request is placed at the top of the backlog it will receive immediate attention, instead of waiting for the next sprint.

Contributors can be individuals, teams, departments, or even entire organizations like an outsourcing firm or a business partner. There is no expectation or requirement that the tasks be done by Scrum teams of 8-10 members. This allows organizations to call on the talents of all kinds of individuals and companies, and in fact conforms to the reality of most large projects today.

Tasks are then managed using Kanban or lean principles. Typically this means that each person is working on one task at a time (i.e., the team has a work-in-process limit of one task for each person on the team).

Kanban principles ensure that once tasks are started they are completed as quickly as possible, which means that they can be released sooner, and also that other tasks which depend on the first task can be started sooner.

When tasks are completed, the contributor pulls in on-demand resources to build and test a release with the new code. This provides immediate feedback to the contributors, and allows them to catch and fix bugs right away. It also makes features available faster, because there is no wait for centralized build and test systems.

Finally, once new code submissions have been tested successfully, they can be pulled through a merge process into a staging area or into a final build. This means that a new version of the software can be assembled and released at any time, with whatever bug fixes and enhancements are available at that moment.

What Does This Accomplish?
How exactly does this Scalable Agile process framework address the shortcomings of Scrum and provide more scalable, responsive development efforts? Here are a few of the advantages:

  • There can be many types of contributors, including (but not limited to) conventional Scrum teams.
  • There is no need to spend time estimating tasks precisely, doing detailed sprint planning, or having long meetings to coordinate assignments across teams. As long as the backlog is maintained in priority order the highest-priority tasks will be addressed first.
  • Once tasks are started they are completed in the least possible time, meaning they can be released faster and dependent tasks can be started sooner.
  • Software quality is better, because test feedback is available as soon as a task is complete. Bugs can be fixed when it is clear what changes caused the problem, and when the code is fresh in the mind of the developer. Also, quality assurance does not become a bottleneck, a situation which often leads organizations to cut corners on testing (leading to yet more quality problems).
  • New versions of the application can be assembled and released at any time according to business demand. With sufficient automation this can be daily or even several times a day.

Scrum is the most popular Agile techniques, but it doesn’t scale well. And while Scrum improves the effectiveness of individual teams, productivity gains fall off sharply on large projects with many teams.

In the earlier section, methods on how to apply agile techniques to distributed teams and large projects was dealt. In the following section,  Tools and techniques for managing distributed teams will be addressed.

Some of the processes and tools needed to manage the Scalable Agile process framework are addressed in the following paragraphs.

The first of these “building blocks” is support for distributed teams. Large development organizations are almost always distributed because they have (1) business units and business partners in multiple locations, (2) “outsourcing” groups in different countries, (3) decided to take advantage of the global market for development talent, (4) remote employees who work from home.

So how can organizations support distributed teams well enough to reduce the need for face-to-face meetings?

Online Agile Planning
Online tools can replace paper-and-pencil planning exercises and physical whiteboards. This allows team members worldwide to create and maintain an overall project backlog, pull tasks to individual teams and contributors, move tasks through the steps in a Kanban process, and view tasks ready to be pulled into a release.

The  an online Agile planning tool is used for managing a central backlog and pulling tasks into “current” task buckets for individual teams, and an online card wall that can replace the physical variety.

online planning tools
Online planning tools can replace paper plans and physical white boards.

Online Collaboration

Development team members can collaborate most easily when they are in the same room, but online tools can provide a close approximation. Such tools include online standup reports, wikis, chat and IM products, and video and teleconferencing systems.

Another type of online tool, an activity stream gives developers real-time visibility into the activities of other team members—activities like code commits, new tickets, comments added to tickets, code reviews and posts on wikis.

activity stream

An activity stream shows commits, comments, and other events.

Global Code Management

Global collaboration can be undermined if developers need to share large repositories and large files over long distances and performance is slow.

Some of the technologies that Tool vendor Perforce uses, like proxies and replication, ensure that files are available immediately in remote locations . These solutions ensure that data is available where needed without artificial boundaries that impede sharing and collaboration.

Perforce technologies ensure that distributed team members don’t have to wait to get large repositories and files.

Decentralized Code Management
Developers often want highly decentralized code management so they can create their own local test branches and work independent of centralized corporate resources.

Development managers, however, want to maintain control over and visibility into activities at remote locations.

Git Fusion from Perforce answers both needs. Developers can quickly clone their own repositories and work in private Git repositories on their local systems, with easy code sharing between teams and products.

select directories

Release managers can make selected directories visible to Git users.

Release managers can model an entire product development effort with Perforce streams and branches, apply access controls, and control how much history and which files are cloned into new Git repositories . As changes are accepted, the enterprise release model guides changes to the right places: older releases, customizations, and parallel development efforts.

When developers commit code to the Perforce repository, the Perforce shared versioning service makes the changes visible to everyone and maintains a strong system of record about the source and nature of all changes.

The earlier paragraphs described the challenges of scaling large and distributed Agile teams, and investigated the tools and strategies that resolve them. Once the problem of scaling Agile development has been addressed, however, pressure is then applied next to the people and processes that are tasked with delivering or deploying the resulting product. Agile workflow is only successful once efficiency is attained in all stages of the workflow. This paragraph will cover the second building block of Scalable Agile: Continuous Delivery.

ScrumBan

ScrumBan is a relatively easy first step for Scrum teams that want to move in the direction of Continuous Delivery.

Teams using ScrumBan work within a time-boxed sprint. But unlike conventional Scrum practices, a work-in-process limit is adopted, so team members are focused on finishing one task at a time. At a certain point in the sprint a “triage” process identifies which tasks can be completed within the time box, and drops the others from the sprint plan. At that point there is a “feature freeze,” and the remainder of the sprint is devoted to completing the tasks specified by the triage process.

scrumban

ScrumBan represents a step toward Continuous Delivery because it emphasizes completing a small number of tasks as quickly as possible. Development teams avoid the pitfalls involved in pulling out all the stops to deliver the entire sprint plan, regardless of the cost in terms of quality and delays.

On-Demand Merge and Test by Contributor
The conventional software release process creates a huge bottleneck at the test phase. All teams send their contributions to a central QA team, which creates and tests a “release candidate.”

In traditional release processes, the QA lab becomes a serious bottleneck.

In theory this workflow makes very efficient use of the QA team and test systems, however:

  • It takes a long time to run all of the tests.
  • It is hard to debug and troubleshoot many code changes at once, especially if they may be interacting with each other.
  • Errors uncovered during the integration phase may require costly rework by several contributors.
  • The test lab becomes a huge bottleneck near the end of each sprint, causing stress and leading to sloppy testing practices.
  • Releases are delayed until the entire release candidate has been completely tested and debugged.

But what if each team can build and test based on just its own contributions?

There can be a different approach to testing. In this scenario each team and contributor has access to test resources. QA team members act as advisors and facilitators rather than being charged with managing all of the testing themselves. When a development team finishes a set of changes, the team then pulls a copy of the production version onto the test system. Changes can then be merged into this private production version, built and tested locally.
Each team pulls a product version, merges its changes, and performs its own tests.

team pulls

If testing uncovers problems, these can be solved right away by development, without worrying about interactions with changes from other teams. Multiple teams and contributors can now test and debug contributions independently, and submit them to the central staging area when ready
Teams submit tested contributions when they are ready; releases can be assembled at any time.

The advantages of this approach include:

  • QA is no longer a bottleneck—teams test independently, when they are ready.
  • It is easier to debug and troubleshoot problems, because each group is observing only its own changes to a previously tested production version.
  • Releases can be assembled at any time, constructed from whatever contributions have been tested and submitted.

These capabilities are not easy or cheap to implement. They require a considerable investment in automated build and test environments, which for distributed teams must be provided in the cloud. They also require that code merge and management be a fast and easy part of any developer’s daily work. A simple and easy-to-automate merge framework like Perforce Streams  provides merge notifications, merge pathway guidance, and intuitive tools.

In a complicated project consisting of several components, Perforce’s visibility over any part of the project also helps development teams share and reuse code. These teams can quickly adapt to a changing project structure, even if they are working in distributed repositories via Git Fusion. A merge in this environment would never require a complicated action that spans several independent repositories.

These capabilities are an indispensable aspect of Scalable Agile, because they allow very large numbers of teams to contribute to a project without overwhelming build and test resources.

Code Review Workflow
Another major challenge for Continuous Delivery is how to merge a growing number of contributions into production releases. How can you organize the flow of contributions from many sources? How can you decide when to assemble the next release? How do you avoid creating a bottleneck at the point where the contributions come together?

One very useful method is a code review workflow similar to those used in open source projects. In these projects hundreds of contributors might submit code and thousands might test it. Typically a core group of “maintainers” reviews submissions and selects the ones that will be included in the next release.

A code review workflow can be utilized in commercial environments as well. for example, the Assembla ticketing tool includes a merge request feature that allows contributors to submit code changes for review. Designated reviewers can review the submissions, hold online discussions about them, vote for or against accepting them, and make immediate decisions to accept or reject them

This code review workflow lets organizations manage the code review process and delegate the decision making for accepting contributions and assembling releases, which prevents these activities from becoming bottlenecks.

A code review workflow allows designated reviewers to vote on which contributions to include in the next release.

Streams for Managing Multiple Versions
Another common challenge among large projects is maintaining multiple releases and managing custom versions for individual customers.

Software vendors, for example, usually need to support several releases of an application at once. Bug fixes might need to be applied to many (but not all) of the supported releases. Enhancements to the current release might be retrofitted to the previous release and added to the upcoming release under development. Similarly, a service provider or enterprise IT department might be maintaining customized versions of an application for different customers or different business units within the enterprise.

perforce streams

It is much easier to navigate these complex scenarios with a tool like Perforce Streams. Perforce Streams not only helps development managers visualize the relationships between releases and versions, it guides release managers on where and when to apply bug fixes and feature enhancements when they are ready to be merged

Perforce Streams provide adaptable workflow for teams and promote efficiencies such as code re-use, automated merging, fast context switching, efficient workspace updates, and inherited workspace and branch views. An innovative addition to the Perforce branching and merging toolset, streams eliminate overhead, simplify common processes, and increase agility and scalability. In projects with a large volume of data, the time and performance savings are considerable.

Perforce Streams helps deploy bug fixes and enhancement across multiple releases and custom versions.

The typical perception of Agile development methodologies is that their benefits and promise are reserved for small, co-located teams. However, in the above sections we have seen how many, if not all, of the traditional Agile practices can be improved to the benefit of not only large teams, but large distributed teams as well. Ironically, this scalability has been achieved by employing the very processes and tools that the Agile Manifesto preaches against. However, while the tools enable scalability, they never require the sacrifice of developers’ freedoms or their ability to interact.

In these final paragraphs , tools will again be the focus for providing the solution for scaling one of the essential requirements of any Agile workflow: Continuous Integration with build and test automation. All the ideas addressed in the above paragraphs  will be reviewed along with detail on  how they fit together to make Agile scalable.

All of the examples provided can be implemented using software from Perforce, Assembla, Git and Jenkins.

Methods for Providing On-Demand Infrastructure

In a large project, the trickiest and costliest problems are found only when individuals put together all the pieces. It pays to find and fix these integration problems as early and often as possible.
Continuous Integration is a set of best practices in software development that supports project integration on a rapid, repeated basis . They are:

  •  Maintain a code repository
  •  Automate the build
  •  Make the build self-testing
  •  Everyone commits to the mainline every day
  •  Every commit (to the mainline) should be built
  •  Keep the build fast
  •  Test in a clone of the production environment
  •  Make it easy to get the latest deliverables
  •  Everyone can see the results of the latest build
  •  Automate deployment

The goal of is to perform a project-wide integration as often as possible. Striving to achieve this shapes your infrastructure, development practices, and attitudes.
Attitude is important. The entire team must commit to achieving successful integration at all stages of the project. For instance, a development task is not considered to be “done” until the feature appears in the integration build and is proven to work there. That shared commitment should make developers uneasy when they risk divergence by working in isolation for any lengthy period, e.g. when they are using an old build as a stable development base, or when they commit their changes their only infrequently. We cannot emphasize enough the importance of frequent integration. It really does reduce project risk.

Continuous Integration Tools in the Cloud
Organizations clearly need to invest in automated build and test processes if they want to scale up and deliver features faster and release more frequently. This investment can be expensive, but manual methods are obviously not scalable. Also, automated build and test processes tend to produce much higher software quality.

And if teams and contributors are highly distributed? Then the build and test tools must be accessible online, in the cloud.

Automated test tools like Jenkins can be integrated into the code review and merge workflows described earlier paragraphs. Whenever a contribution is accepted, a new version can be built and a series of automated tests can be run against it. Tools like Jenkins will then provide developers and the QA staff with detailed information on test results. Results from the test tool can even be used to vote to accept or reject contributions, as part of the code review workflow.

Automated test tools can provide detailed information on test results, and even vote to accept or reject contributions.

jenkins ci

Managing the impact of on-demand continuous integration is a logistical challenge for the version management service. Perforce addresses this challenge by providing flexible configurations of proxies and replicas to meet a variety of build demands.

supporting ci

In the earlier paragraphs,  some of the shortcomings of Scrum, such as lack of techniques to coordinate teams, assumptions about co-located teams and fixed release cycles that are unrealistic for many organizations, and a tendency to spend too much time in planning and coordination meetings was covered.

Continuous, iterative development is supported for different workflow methodology employed. With Perforce,  can:

  • Build and confirm your work from a private workspace before submitting your code
  • Execute automated builds and tests on specific branches upon check-in
  • Improve the quality of software and delivery time to market
  • Popular continuous integration tools, like Electric Commander from Electric Cloud, Parabuild from Viewtier, and Anthill Pro from UrbanCode, all support their own integrations with Perforce.

 A Scalable Agile process framework featuring the following was outlined:

  • A single prioritized backlog for all teams, so high-priority tasks always receive immediate attention.
  • Kanban processes with WIP limits, so teams don’t have to spend a lot of time in release planning, and to ensure that individual tasks are completed as quickly as possible.
  • On-demand resources, so each team can build and test its own contributions quickly and avoid making the QA lab a bottleneck.
  • A code review process that allows designated reviewers to accept or reject a large number of code submissions.
  • A “take what’s ready” approach to releases, so organizations could provide new functionality as frequently as required by customer needs and expectations.

The processes and tools that could facilitate Scalable Agile was discussed. These include online Agile planning tools, online collaboration, global code management, decentralized code management, ScrumBan processes, tools for on-demand merging and testing by contributors, code review workflows, stream-based tools for managing multiple releases and custom versions, and continuous integration tools provided in the cloud.

While the journey to Scalable Agile may be a long one, each of the steps down the path provides immediate benefits. The growing development groups consider:

  • Implementing ScrumBan, to start moving toward lean methods.
  • Deploying online planning and collaboration tools, to improve the effectiveness of distributed teams and contributors.
  • Deploying advanced code management platforms, to support distributed development and manage multiple releases and versions.
  • Begin investing in Continuous Integration and on-demand build and test systems.
  • Adjust the dial on continuous delivery gradually, to allow time for all your teams to adjust.
Advertisements

PEARL XVI : SCRUMBAN – A tool for software production and Support

PEARL XVI : Scrumban is a combination between Scrum and Kanban methodologies for increased applicability and versatility tool for software production and support focused companies.

Scrum is an iterative and prescriptive process for building software using the Agile methodology. A development team plans and commits to completing a certain amount of work in a certain time period called a sprint. At the end of the sprint, the team reviews the work with a product owner. They then hold a retrospective to analyze their processes during the sprint, and determine what can be improved next time.

Scrum can be an effective tool for introducing teams to Agile. Its more rigid structure provides a framework of understanding that is far easier to grasp than the loose nature of Kanban might be for teams used to rigidly planned waterfall-style projects.

The daily standup, planning, review, and retrospective meetings are excellent touch points for periodic checking in with the work that is happening and reviewing with stakeholders. The retrospective itself is the crown jewel of the Scrum framework, and is what enables teams to focus on continuous improvement, or Kaizen.

One of the primary motivations for moving a team to Scrum is to get away from the often restrictive and inefficient processes of the waterfall model. In this model, there is frequently too much time spent on planning and designing before any work begins. After that, teams find themselves unable to respond to change later in the project as the developing and testing proceeds. All of this is why waterfall is a sub-optimal process for software development.

But look at the makeup of a typical Scrum sprint:

  1. Team plans the work that will be worked on over the next sprint.
  2. During planning, teams try to design as many features as possible, so that they can more accurately estimate what they can complete during the Sprint.
  3. During the Sprint, the team develops and then tests their user stories.
  4. At the end of the Sprint, the Product Owner reviews the work completed, and decides which of the stories are shippable and ready for production.

What does this mean? Scrum is a series of miniature waterfall projects wrapped up into iterations called “Sprints.” This is a push-based system, where another solution would be to use a pull-based one. Enter Scrumban.

Scrumban can give teams the power to adapt and change to stakeholder and production needs, without feeling overburdened by their project methodology. It is designed to remove metrics that encourage undesired outcomes. It can restore working time to the team, and avoids unnecessary meetings. And most importantly, it can limit the team’s work in progress so that they can finish what they start to a high standard. Scrumban can remove overhead stress for the development team, increase efficiency, and increase the overall satisfaction for the customer.

Scrumban combines the best features of both methods. It compounds the prescriptive nature of Scrum and the process improvement of Kanban, allowing teams to become Agile and to continually improve their process. Scrumban is becoming especially popular in services industries, where project development and maintenance goes together.

Scrumban is a process for evolving a Scrum instance to be more Lean oriented, supplemented with core Kanban practices such as: visualization, work-in-process limits, work flow management, and making policies explicit. The basic Scrumban implementation principles include:

  • Start with all the ceremonies, artifacts and roles you use now.
  • Agree to pursue improvement towards a more effective process.
  • Respect current roles, responsibilities and job titles.

Scrumban as a combination of a Kanban and Scrum method

Scrum-ban is a software production model based on Scrum and Kanban. Scrum-ban is especially suited for maintenance projects or (system) projects with frequent and unexpected work items or programming errors. In such cases the time-limited sprints of the Scrum model are of no appreciable use, but Scrum’s daily meetings and other practices can be applied, depending on the team and the situation at hand. Visualization of the work stages and limitations for simultaneous unfinished work and defects are familiar from the Kanban model. Using these methods, the team’s workflow is directed in a way that allows for minimum completion time for each work item or programming error, and on the other hand ensures each team member is constantly employed.

To illustrate each stage of work, teams working in the same space often use post-it notes or a large whiteboard. In the case of decentralized teams, stage-illustration software such as Assembla, ScrumWorks, TargetProcess, Rational Team Concert, Eylean board, OnTime, Kanban Tool or JIRA in combination with Jira Agile can be used to visualize each team’s product backlog items, defects and tasks divided into separate phases.

In their simplest, the tasks are categorized into the work stages:

  • Unstarted
  • Ongoing
  • Completed

If desired, though, the teams can add more stages of work (such as “defined”, “designed”, “tested” or “delivered”). These additional phases can be of assistance if a certain part of the work becomes a bottleneck and the limiting values of the unfinished work cannot be raised. A more specific task division also makes it possible for employees to specialize in a certain phase of work.

There are no set limiting values for unfinished work. Instead, each team has to define them individually by trial and error; a value too small results in workers standing idle for lack of work, whereas values too high tend to accumulate large amounts of unfinished work, which in turn hinders completion times. A rule of thumb worth bearing in mind is that no team member should have more than two simultaneous selected tasks, and that on the other hand not all team members should have two tasks simultaneously.

The major differences between Scrum and Kanban are derived from the fact that, in Scrum, work is divided into sprints that last a certain amount of time, whereas in Kanban the workflow is continuous. This is visible in work stage tables, which in Scrum are emptied after each sprint. In Kanban all tasks are marked on the same table. Scrum focuses on teams with multifaceted know-how, whereas Kanban makes specialized, functional teams possible.

In Kanban, Workitems deemed  high priority and needed as soon as  possible have a way of being addressed  ahead of current work. Periodic reviews  of process, tools, team performance, and
WIP limits to find improvements and  eliminate inefficiencies or waste.

When to use Scrumban?

Are you considering implementing Scrumban in your team or business? You should take it into account if you are doing one of these:

  • Projects maintenance,
  • Event-driven work (support, hardening),
  • Problematic projects management (projects with unexpected user stories and bugs),
  • New product development (work preceding sprint development or following sprint development),
  • Continuous management improvement.

Scrumban board gives an easy view of process workflow. It informs about the number of items the team is currently working on as well as already finished ones. It influences the improvement of accountability, responsibility, communication and performance results.

 The example of a Scrumban board

Scrumban is a pull-based system, where the team no longer plans out the work that is committed to during the planning meeting, and instead continually grooms the backlog. The same Scrum meetings (planning, review, and retrospective) can and should still take place, but the cadence of them can be more context-driven. The real key to moving to Scrumban, though, is ensuring that work in progress (WIP) is still limited.

Work-in-progress limits, not Sprints. With Scrum, the amount of work that is ongoing is limited by the Sprint time commitment. But in Scrumban, with no specific time commitment, the team must limit itself through the use of WIP limits on columns within their task board. The goal is always to move tickets in a flow from left to right on the board. If too many issues are in progress, the team is at risk of not finishing anything to high quality standards. Instead, there should be a maximum number of tickets allowed per column. If the number of tickets in that column ever exceeds the maximum, the entire team should swarm onto that column and help move tickets on. This should happen no matter what functional role a team member fills.

Key Scrumban Terms
• Work Backlog – Equivalent to Scrum’s Product Backlog with minor differences for how  Workitems are tracked.
• Workitem– The individual descriptions of functionality needed by the business that either define enhancements or defects.
• Workline – The series of stages in the software development process that pull work from the  Work Backlog and produce shippable business value.
• Stages – The common and discrete steps required to deliver software within an organization.

The Weekly Timeblock (WTB)

The Weekly Timeblock is a reoccurring meeting used for multiple purposes. It should be set up in the  middle of the week (Wednesday are best) so as to avoid holidays and vacation days that typically  occur at the beginning and end of each workweek.
Two to four hours is recommended and set to occur in the afternoon so any preparation work can be  completed ahead of time. The team will use the time they need and leave the rest. Having a projector,  screen, and ample whiteboard space will be conducive for any technical discussions and problem  solving.
The first use of the Weekly Timeblock is for demonstrating any completed work. This is the time  where Developers can show the Product Owner what was completed and get Acceptance. Completed  work can be demonstrated outside of the meeting anytime but it is good for the team to have  every one witness the conclusion of work.
Once Workitems have been demonstrated, the team should reflect on how their development process  is going, any tools they use, and review the data on how Workitems are flowing through the  Workline. They should have frank discussions about any new innovation that should be reinforced or  impediments to be overcome. Clear action items and their owners should be identified along with a  target resolution date.
After the review of recent activity, the team should have a preview of upcoming Workitems the  Product Owner is preparing. The Developers should seek to understand the intent of each item,  asking questions and getting clarity so everyone is familiar with what’s needed. The Developers  should also provide feedback on Workitem scope, structure, and the optimal execution order so the  Product Owner can make any needed adjustments to content and priority.

Standup meetings. The Daily Standup meeting is a concept borrowed from Scrum and serves the same purpose; to coordinate activities and identify impediments. A daily reoccurring meeting should be set up for 30  minutes where the first half is focused on the team members updating everyone on what they did  since the last meeting, what they intend to do before the next meeting, and any impediments that are  blocking, or threaten to block progress on their work. The second half is for any impromptu  discussions necessary to address questions or plan activities. The meeting should be held near the Board so discussions focus around work in progress and updates can be made publically. Ideally the  meeting should occur early in the day, just after everyone has had a chance to get to work, reorient themselves, and think about what they’ll focus on.

In practice, this boils down to redundant statuses that recount information available on the team’s task board. For Scrumban, a more effective method is to refocus on the flow of tickets on the board. That same pattern of yesterday/today/blocked can be transferred to the tickets themselves—the group moves through each column and briefly discusses each ticket and what is necessary to move that ticket rightward on the board. This provides far more context to the team and informs everyone of any major architectural or design decisions.

Metrics. Metrics can certainly be useful, but they are often abused by managers and business stakeholders who want to unnaturally simplify a complex process into a one-dimensional number. Velocity, the amount of story points a Scrum team completes in a single Sprint, is such a metric that incentivizes lower quality at the end of a Sprint as a team scrambles to finish every last story they committed to. When the number fluctuates, as is common with a newer team, the stakeholders begin to question the outputs of the team, and even the effectiveness of Agile itself.

Instead of velocity, a useful Scrumban metric is cycle time. This is the length of time a ticket takes to complete, measured from when it is first began. Over time, a statistical analysis of all tickets in the project can yield a mean cycle time and standard deviation. This can be a useful planning tool at a macro level, as it is trivial to add up the number of stories and multiply by mean cycle time.

The Work Backlog under Scrumban is basically the same as with Scrum. It is a collection of User  Stories and Acceptance Criteria written and maintained by the Product Owner. These individual Workitems describe the business value desired.

To keep the collection organized, additional information is associated with each item:
• ID–A unique identifier (e.g. B#042) for easy searches and reference.
• Project –A label to group a set of work items together according to a theme or purpose.
• Title –A concise, action-oriented description of the business value desired.
• User Story –A formulaic description of who will benefit from the item, what exactly is  wanted, and why it is wanted. The User Story provides the context behind the request and  enables the Developers to understand the goal, and not just take orders.
• Acceptance Criteria –A list of the specific conditions required for the Workitem to be  considered complete.
• Status –An indicator of the readiness of the Workitem, when it is being worked on, and the  final disposition.

When the progress of a Work item through the Workline is blocked

As a team is working on items, it will be fairly common that progress from one stage to another is blocked. These blockages can be internalto the team where the work was more challenging than expected. Whatever the cause the team should “swarm” to help wherever possible to remove the blockage. All team members can assist to one degree or another, even crossing the traditional role boundaries of Coders & Testers to help test, code, prepare environments, research or write documentation.
When a blocking issue is external to the team, such as when a required server hasn’t yet been delivered or subject matter experts are needed for questions, the Scrum Master should attempt to resolve it. The rest of the team can be engaged as needed to help with external issues.
If a Workitem is blocked completely with no resolution in sight, it should be pulled from the Workline and its status changed from ‘In Progress’ to ‘New’ or ‘Draft’ depending on what is needed before the team can pull it back into the Workline again.

Workitem needs to be completed as soon as possible
There are two ways to address this need. First, the Product Owner should be continually maintaining  the order of the Work Backlog such that the items the business considers the most valuable are at or  near the top. It is these top items that the team reviews and makes ‘Ready’ through discussions and  feedback and are pulled into the Workline as capacity opens up.
If there a more urgent need for a work item to be completed than what the standard process  provides, the item’s priority can be changed to expedited and be fast-tracked through to completion.

Defects in Scrumban

Defects are a natural part of any software development effort and dealing with them is straightforward. When defects associated with current Workitems are found they should be addressed before the item is considered complete and ready for demonstration.
Assuming the defect was found during the Testing stage and it is relatively minor, then the item can stay within the Testing stage as the Coder works closely with the Testing to diagnose the issue and provide updated code.
If the diagnosing and fixing effort is much larger however, such as requiring more than a day to complete, then the Workitem should move back to the Coding stage even if the WIP limits are exceeded. The priority for the Coder should be to address the defect and get the Workline flowing again in the right direction.
Sometimes defects are found in areas of the code not currently part of a Workitem in progress. These should be brought to the attention of the Product Owner who can create a new Workitem representing the defect and prioritize it accordingly in the Work Backlog.

Estimation

The statistics kept on how long Workitems take to transit the Workline should be enough to estimate  how long any particular item will take to be completed. For example, if the average number of days is  four, then any new item pulled into the Workline is likely to take four days. By the same logic, if there was a set of Workitems that comprised a release, then the time to complete the entire set is four  times the number of items in the set

This estimation technique works well when the scopes of the Workitems are somewhat consistent.  When they are not, some rough estimates of size, combined with the transit time for those specific  sizes can produce a more accurate estimate although it will take longer to gather enough data to be  confident with the results.Be mindful that any additional time spent by the team should be worth the  effort and if not, abandoned for the simple statistics Kanban provides.