PEARL III: Principles of Lean Software Development for Agile Methodology

PEARL III: Principles of Lean Software Development for Agile Methodology

The market for software is fast paced with frequently changing customer needs. In order to stay competitive companies have to be able to react to the changing needs in a rapid manner. Failing to do so often results in a higher risk of market lock-out , reduced probability of market dominance , and it is less likely that the product conforms to the needs of the market. In consequence software companies need to take action in order to be responsive whenever there is a shift in the customers’ needs on the market. That is, they need to meet the current requirements of the market, the requirements being function or quality related. Two development paradigms emerged in the last decade to address this challenge, namely agile and lean software development.

The term Lean Software Development was first coined as the title for a conference organized by the ESPRIT initiative of the European Union, in Stuttgart Germany, October 1992. Independently, the following year, Robert Charette in 1993 suggested the concept of “Lean Software Development” as part of his work exploring better ways of managing risk in software projects. The term “Lean” dates to 1991, suggested by James Womack, Daniel Jones, and Daniel Roos, in their book The Machine That Changed the World: The Story of Lean Production as the English language term to describe the management approach used at Toyota. The idea that Lean might be applicable in software development was established very early, only 1 to 2 years after the term was first used in association with trends in manufacturing processes and industrial engineering.

In their 2nd book, published in 1995, Womack and Jones defined five core pillars of Lean Thinking. These were:

  • Value
  • Value Stream
  • Flow
  • Pull
  • Perfection

This became the default working definition for Lean over most of the next decade. The pursuit of perfection, it was suggested, was achieved by eliminating waste. While there were 5 pillars, it was the 5th one, pursuit of perfection through the systemic identification of wasteful activities and their elimination, that really resonated with a wide audience. Lean became almost exclusively associated with the practice of elimination of waste through the late 1990s and the early part of the 21st Century.

The Womack and Jones definition for Lean is not shared universally. The principles of management at Toyota are far more subtle. The single word “waste” in English is described more richly with three Japanese terms:

  • Muda – literally meaning “waste” but implying non-value-added activity
  • Mura – meaning “unevenness” and interpreted as “variability in flow”
  • Muri – meaning “overburdening” or “unreasonableness”

Perfection is pursued through the reduction of non-value-added activity but also through the smoothing of flow and the elimination of overburdening. In addition, the Toyota approach was based in a foundational respect for people and heavily influenced by the teachings of 20th century quality assurance and statistical process control experts such as W. Edwards Deming.

Unfortunately, there are almost as many definitions for Lean as there are authors on the subject.

Lean and Agile

Bob Charette was invited but unable to attend the 2001 meeting at Snowbird, Utah, where the Manifesto for Agile Software Development was authored. Despite missing this historic meeting, Lean Software Development was considered as one of several Agile approaches to software development. Jim Highsmith dedicated a chapter of his 2002 book to an interview with Bob about the topic. Later, Mary & Tom Poppendieck went on to author a series of 3 books. During the first few years of the 21st Century, Lean principles were used to explain why Agile methods were better. Lean explained that Agile methods contained little “waste” and hence produced a better economic outcome. Lean principles were used as a “permission giver” to adopt Agile methods.
Early lean software ideas were developed by Poppendieck and Middleton and Sutton. These books explored how lean thinking could be transferred from manufacturing to the more intangible world and different culture of software engineers. Specific techniques on how the concept of kanban could be applied to software were also developed . Note
that the use of these methods is partly a metaphor rather than a direct copying. For example, kanban in factories literally is a binary signal to replenish an inventory buffer, based on what the customer has taken away. In software it performs a similar function, but more broadly displays information on the status of the process and potential problems.
Moving upstream and applying lean thinking to influence project selection and definition also creates great benefits .
The proceedings of the first Lean & Kanban Software conference and the work of Shalloway et al. show adoption is spreading.

Defining Lean Software Development

Defining Lean Software Development is challenging because there is no specific Lean Software Development method or process. Lean is not an equivalent of Personal Software Process, V-Model, Spiral Model, EVO, Feature-Driven Development, Extreme Programming, Scrum, or Test-Driven Development. A software development lifecycle process or a project management process could be said to be “lean” if it was observed to be aligned with the values of the Lean Software Development movement and the principles of Lean Software Development. So those anticipating a simple recipe that can be followed and named Lean Software Development will be disappointed. Individuals must fashion or tailor their own software development process by understanding Lean principles and adopting the core values of Lean.

There are several schools of thought within Lean Software Development. The largest, and arguably leading, school is the Lean Systems Society, which includes Donald Reinertsen, Jim Sutton, Alan Shalloway, Bob Charette, Mary Poppendeick, and David J. Anderson. Mary and Tom Poppendieck’s work developed prior to the formation of the Society and its credo stands separately, as does the work of Craig Larman, Bas Vodde , and, most recently, Jim Coplien . This section seeks to be broadly representative of the Lean Systems Society viewpoint as expressed in its credo and to provide a synthesis and summary of their ideas.

Values

The Lean Systems Society published its credo at the 2012 Lean Software & Systems Conference . This was based on a set of values published a year earlier. Those values include:
  • Accept the human condition
  • Accept that complexity & uncertainty are natural to knowledge work
  • Work towards a better Economic Outcome
  • While enabling a better Sociological Outcome
  • Seek, embrace & question ideas from a wide range of disciplines
  • A values-based community enhances the speed & depth of positive change

Accept the Human Condition

Knowledge work such as software development is undertaken by human beings. We humans are inherently complex and, while logical thinkers, we are also led by our emotions and some inherent animalistic traits that can’t reasonably be overcome. Our psychology and neuro-psychology must be taken into account when designing systems or processes within which we work. Our social behavior must also be accommodated. Humans are inherently emotional, social, and tribal, and our behavior changes with fatigue and stress. Successful processes will be those that embrace and accommodate the human condition rather than those that try to deny it and assume logical, machine-like behavior.

Accept that Complexity & Uncertainty are Natural to Knowledge Work

The behavior of customers and markets are unpredictable. The flow of work through a process and a collection of workers is unpredictable. Defects and required rework are unpredictable. There is inherent chance or seemingly random behavior at many levels within software development. The purpose, goals, and scope of projects tend to change while they are being delivered. Some of this uncertainty and variability, though initially unknown, is knowable in the sense that it can be studied and quantified and its risks managed, but some variability is unknowable in advance and cannot be adequately anticipated. As a result, systems of Lean Software Development must be able to react to unfolding events, and the system must be able to adapt to changing circumstances. Hence any Lean Software Development process must exist within a framework that permits adaptation (of the process) to unfolding events.

Work towards a better Economic Outcome

Human activities such as Lean Software Development should be focused on producing a better economic outcome. Capitalism is acceptable when it contributes both to the value of the business and the benefit of the customer. Investors and owners of businesses deserve a return on investment. Employees and workers deserve a fair rate of pay for a fair effort in performing the work. Customers deserve a good product or service that delivers on its promised benefits in exchange for a fair price paid. Better economic outcomes will involve delivery of more value to the customer, at lower cost, while managing the capital deployed by the investors or owners in the most effective way possible.

Enable a better Sociological Outcome

Better economic outcomes should not be delivered at the expense of those performing the work. Creating a workplace that respects people by accepting the human condition and provides systems of work that respect the psychological and sociological nature of people is essential. Creating a great place to do great work is a core value of the Lean Software Development community.

Principles of Lean Software Development for Scaling Agile

The Lean Software & Systems community seems to agree on a few principles that underpin Lean Software Development processes. These are the principles of Lean software development for Agile Methodology.
  • Follow a Systems Thinking & Design Approach
  • Emergent Outcomes can be Influenced by Architecting the Context of a Complex Adaptive System
  • Respect People (as part of the system)
  • Use the Scientific Method (to drive improvements)
  • Encourage Leadership
  • Generate Visibility (into work, workflow, and system operation)
  • Reduce Flow Time
  • Reduce Waste to Improve Efficiency

Follow a Systems Thinking & Design Approach

This is often referred to in Lean literature as “optimize the whole,” which implies that it is the output from the entire system (or process) that we desire to optimize, and we shouldn’t mistakenly optimize parts in the hope that it will magically optimize the whole. Most practitioners believe the corollary to be true, that optimizing parts (local optimization) will lead to a suboptimal outcome.

A Lean Systems Thinking and Design Approach requires that we consider the demands on the system made by external stakeholders, such as customers, and the desired outcome required by those stakeholders. We must study the nature of demand and compare it with the capability of our system to deliver. Demand will include so-called “value demand,” for which customers are willing to pay, and “failure demand,” which is typically rework or additional demand caused by a failure in the supply of value demand. Failure demand often takes two forms: rework on previously delivered value demand and additional services or support due to a failure in supplying value demand. In software development, failure demand is typically requests for bug fixes and requests to a customer care or help desk function.

A systems design approach requires that we also follow the Plan-Do-Study-Act (PDSA) approach to process design and improvement. W. Edwards Deming used the words “study” and “capability” to imply that we study the natural philosophy of our system’s behavior. This system consists of our software development process and all the people operating it. It will have an observable behavior in terms of lead time, quality, quantity of features or functions delivered (referred to in Agile literature as “velocity”), and so forth.

Velocity : At the end of each iteration, the team adds up effort estimates associated with user stories that were completed during that iteration. This total is called velocity.
Knowing velocity, the team can compute (or revise) an estimate of how long the project will take to complete, based on the estimates associated with remaining user stories and assuming that velocity over the remaining iterations will remain approximately the same. This is generally an accurate prediction, even though rarely a precise one.

“Lead time” is a term borrowed from the manufacturing method known as Lean or Toyota Production System, where it is defined as the time elapsed between a customer placing an order and receiving the product ordered.
Translated to the software domain, lead time can be described more abstractly as the time elapsed between the identification of a requirement and its fulfillment. Defining a more concrete measurement depends on the situation being examined: for instance, when focusing on the software development process, the “lead time” elapsed between the formulation of a user story and that story being used “in production”, that is, by actual users under normal conditions.
Teams opting for the kanban approach favor this measure, over the better known velocity. Instead of aiming at increasing velocity, improvement initiatives intend to reduce lead time.

These metrics will exhibit variability and, by studying the mean and spread of variation, Individuals can develop an understanding of their capability. If this is mismatched with the demand and customer expectations, then the system will need to be redesigned to close the gap.

Deming also taught that capability is 95% influenced by system design, and only 5% is contributed by the performance of individuals. In other words, we can respect people by not blaming them for a gap in capability compared to demand and by redesigning the system to enable them to be successful.

To understand system design, we must have a scientific understanding of the dynamics of system capability and how it might be affected. Models are developed to predict the dynamics of the system. While there are many possible models, several popular ones are in common usage: the understanding of economic costs; so-called transaction and coordination costs that relate to production of customer-valued products or services; the Theory of Constraints – the understanding of bottlenecks; and The Theory of Profound Knowledge – the study and recognition of variability as either common to the system design or special and external to the system design.

Emergent Outcomes can be Influenced by ARCHITECTURE of the Context for a Complex Adaptive System

Complex systems have starting conditions and simple rules that, when run iteratively, produce an emergent outcome. Emergent outcomes are difficult or impossible to predict given the starting conditions. The computer science experiment “The Game of Life” is an example of a complex system. A complex adaptive system has within it some self-awareness and an internal method of reflection that enables it to consider how well its current set of rules is enabling it to achieve a desired outcome. The complex adaptive system may then choose to adapt itself – to change its simple rules – to close the gap between the current outcome and the desired outcome. The Game of Life adapted such that the rules could be re-written during play would be a complex adaptive system.

In software development processes, the “simple rules” of complex adaptive systems are the policies that make up the process definition. The core principle here is based in the belief that developing software products and services is not a deterministic activity, and hence a defined process that cannot adapt itself will not be an adequate response to unforeseeable events. Hence, the process designed as part of our system thinking and design approach must be adaptable. It adapts through the modification of the policies of which it is made.

The Kanban approach to Lean Software Development utilizes this concept by treating the policies of the kanban pull system as the “simple rules,” and the starting conditions are that work and workflow is visualized, that flow is managed using an understanding of system dynamics, and that the organization uses a scientific approach to understanding, proposing, and implementing process improvements.

The term “kanban” with the sense of a sign, poster or billboard, and derived from roots which literally translate as “visual board”.
Its meaning within the Agile context is borrowed from the Toyota Production System, where it designates a system to control the inventory levels of various parts. It is analogous to (and in fact inspired by) cards placed behind products on supermarket shelves to signal “out of stock” items and trigger a resupply “just in time”.
The Toyota system affords a precise accounting of inventory or “work in process”, and strives for a reduction of inventory levels, considered wasteful and harmful to performance.
The phrase “Kanban method” also refers to an approach to continuous improvement which relies on visualizing the current system of work scheduling, managing “flow” as the primary measure of performance, and whole-system optimization – as a process improvement approach, it does not prescribe any particular practices.

Respect People

The Lean community adopts Peter Drucker’s definition of knowledge work that states that workers are knowledge workers if they are more knowledgeable about the work they perform than their bosses. This creates the implication that workers are best placed to make decisions about how to perform work and how to modify processes to improve how work is performed. So the voice of the worker should be respected. Workers should be empowered to self-organize to complete work and achieve desired outcomes. They should also be empowered to suggest and implement process improvement opportunities or “kaizen events” as they are referred to in Lean literature. Making process policies explicit so that workers are aware of the rules that constrain them is another way of respecting them. Clearly defined rules encourage self-organization by removing fear and the need for courage. Respecting people by empowering them and giving them a set of explicitly declared policies holds true with the core value of respecting the human condition.
SAP has been using SCRUM and other Agile methodologies for several years at the team level. Herbert Illgner, COO Business Solutions and Technology at SAP, who has been involved with the effort, says that team empowerment and faster feedback cycles with customers are two significant benefits. Illgner added that SAP is expanding application of Agile methods to the entire product creation process using a Lean framework that includes empowered cross-functional teams, continuous improvement process and managers as support and teachers..

Use the Scientific Method

Seek to use models to understand the dynamics of how work is done and how the system of Lean Software Development is operating. Observe and study the system and its capability, and then develop and apply models for predicting its behavior. Collect quantitative data in the applicable studies, and use that data to understand how the system is performing and to predict how it might change when the process is changed.

The Lean Software & Systems community uses statistical methods such as statistical process control charts and spectral analysis histograms of raw data for lead time and velocity to understand system capability. They also use models such as: the Theory of Constraints to understand bottlenecks; The System of Profound Knowledge to understand variation that is internal to the system design versus that which is externally influenced; and an analysis of economic costs in the form of tasks performed to merely coordinate, set up, deliver, or clean up after customer-valued product or services are created. Some other models are coming into use, such as Real Option Theory, which seeks to apply financial option theory from financial risk management to real-world decision making.

The scientific method suggests: we study; we postulate an outcome based on a model; we perturb the system based on that prediction; and we observe again to see if the perturbation produced the results the model predicted. If it doesn’t, then we check our data and reconsider whether our model is accurate. Using models to drive process improvements moves it to a scientific activity and elevates it from a superstitious activity based on intuition.

Encourage Leadership

Leadership and management are not the same. Management is the activity of designing processes, creating, modifying, and deleting policy, making strategic and operational decisions, gathering resources, providing finance and facilities, and communicating information about context such as strategy, goals, and desired outcomes. Leadership is about vision, strategy, tactics, courage, innovation, judgment, advocacy, and many more attributes. Leadership can and should come from anyone within an organization. Small acts of leadership from workers will create a cascade of improvements that will deliver the changes needed to create a Lean Software Development process.

Generate Visibility

Knowledge work is invisible. If you can’t see something, it is (almost) impossible to manage it. It is necessary to generate visibility into the work being undertaken and the flow of that work through a network of individuals, skills, and departments until it is complete. It is necessary to create visibility into the process design by finding ways of visualizing the flow of the process and by making the policies of the process explicit for everyone to see and consider. When all of these  are visible, then the use of the scientific method is possible, and conversations about potential improvements can be collaborative and objective. Collaborative process improvement is almost impossible if work and workflow are invisible and if process policies are not explicit.

Reduce Flow Time

The software development profession and the academics who study software engineering have traditionally focused on measuring time spent working on an activity. The Lean Software Development community has discovered that it might be more useful to measure the actual elapsed calendar time something takes to be processed. This is typically referred to as Cycle Time and is usually qualified by the boundaries of the activities performed. For example, Cycle Time through Analysis to Ready for Deployment would measure the total elapsed time for a work item, such as a user story, to be analyzed, designed, developed, tested in several ways, and queued ready for deployment to a production environment. In consultation with the customer or product owner, the team divides up the work to be done into functional increments called “user stories”.

Lead time clock starts when the request is made and ends at delivery. Cycle time clock starts when work begins on the request and ends when the item is ready for delivery. Cycle time is a more mechanical measure of process capability. Lead time is what the customer sees.

Lead time depends on cycle time, but also depends on your willingness to keep a backlog, the customer’s patience, and the customer’s readiness for delivery.

Focusing on the time work takes to flow through the process is important in several ways. Longer cycle times have been shown to correlate with a non-linear growth in bug rates. Hence shorter cycle times lead to higher quality. This is counter-intuitive as it seems ridiculous that bugs could be inserted in code while it is queuing and no human is actually touching it. Traditionally, the software engineering profession and academics who study it have ignored this idle time. However, empirical evidence suggests that cycle time is important to initial quality.

Alan Shalloway has also talked about the concept of “induced work.” His observation is that a lag in performing a task can lead to that task taking a lot more effort than it may have done. For example, a bug found and fixed immediately may only take 20 minutes to fix, but if that bug is triaged, is queued and then waits for several days or weeks to be fixed, it may involve several or many hours to make the fix. Hence, the cycle time delay has “induced” additional work. As this work is avoidable, in Lean terms, it must be seen as “waste.”

The third reason for focusing on cycle time is a business related reason. Every feature, function, or user story has a value. That value may be uncertain but, nevertheless, there is a value. The value may vary over time. The concept of value varying over time can be expressed economically as a market payoff function. When the market payoff function for a work item is understood, even if the function exhibits a spread of values to model uncertainty, it is possible to evaluate a “cost of delay.” The cost of delay allows us to put a value on reducing cycle time.

With some work items, the market payoff function does not start until a known date in the future. For example, a feature designed to be used during the 4th of July holiday in the United States has no value prior to that date. Shortening cycle time and being capable of predicting cycle time with some certainty is still useful in such an example. Ideally, we want to start the work so that the feature is delivered “just in time” when it is needed and not significantly prior to the desired date, nor late, as late delivery incurs a cost of delay. Just-in-time delivery ensures that optimal use was made of available resources. Early delivery implies that we might have worked on something else and have, by implication, incurred an opportunity cost of delay.

As a result of these three reasons, Lean Software Development seeks to minimize flow time and to record data that enables predictions about flow time. The objective is to minimize failure demand from bugs, waste from over-burdening due to delay in fixing bugs, and to maximize value delivered by avoiding both cost of delay and opportunity cost of delay.

Reduce Waste to Improve Efficiency

A value stream mapping technique is used to identify waste. The second step is to point out sources of waste and to eliminate them. Waste-removal should take place iteratively until even essential-seeming processes and procedures are liquidated.

For every valued-added activity, there are setup, cleanup and delivery activities that are necessary but do not add value in their own right. For example, a project iteration that develops an increment of working software requires planning (a setup activity), an environment and perhaps a code branch in version control (collectively known as configuration management and also a setup activity), a release plan and performing the actual release (a delivery activity), a demonstration to the customer (a delivery activity), and perhaps an environment teardown or reconfiguration (a cleanup activity.) In economic terms, the setup, cleanup, and delivery activities are transaction costs on performing the value-added work. These costs (or overheads) are considered waste in Lean.

Any form of communication overhead can be considered waste. Meetings to determine project status and to schedule or assign work to team members would be considered a coordination cost in economic language. All coordination costs are waste in Lean thinking. Lean software development methods seek to eliminate or reduce coordination costs through the use of colocation of team members, short face-to-face meetings such as standups, and visual controls such as card walls.

The third common form of waste in Lean Software Development is failure demand. Failure demand is a burden on the system of software development. Failure demand is typically rework or new forms of work generated as a side-effect of poor quality. The most typical forms of failure demand in software development are bugs, production defects, and customer support activities driven out of a failure to use the software as intended. The percentage of work-in-progress that is failure demand is often referred to as Failure Load. The percentage of value-adding work against failure demand is a measure of the efficiency of the system.

The percentage of value-added work against the total work, including all the non-value adding transaction and coordination costs, determines the level of efficiency. A system with no transaction and coordination costs and no failure load would be considered 100% efficient.

Traditionally, Western management science has taught that efficiency can be improved by increasing the batch size of work. Typically, transaction and coordination costs are fixed or rise only slightly with an increase in batch size. As a result, large batches of work are more efficient. This concept is known as “economy of scale.” However, in knowledge work problems, coordination costs tend to rise non-linearly with batch size, while transaction costs can often exhibit a linear growth. As a result, the traditional 20th Century approach to efficiency is not appropriate for knowledge work problems like software development.

It is better to focus on reducing the overheads while keeping batch sizes small in order to improve efficiency. Hence, the Lean way to be efficient is to reduce waste. Lean software development methods focus on fast, cheap, and quick planning methods; low communication overhead; and effective low overhead coordination mechanisms, such as visual controls in kanban systems. They also encourage automated testing and automated deployment to reduce the transaction costs of delivery. Modern tools for minimizing the costs of environment setup and teardown, such as modern version control systems and use of virtualization, also help to improve efficiency of small batches of software development.

Lean Software Development Practices for Agile 

Lean software development are viewed as a set of thinking tools that could easily blend in with any agile approach.So as you can see, lean and agile are deeply intertwined in the software world.
In practice, Agile seems to be changing for the better by adopting Lean thinking in a large way. Rally Development says that its customers get to market 50% faster and are 25% more productive when they employ a hybrid of Lean and Agile development methods.

Lean Software Development does not prescribe practices. It is more important to demonstrate that actual process definitions are aligned with the principles and values. However, a number of practices are being commonly adopted. This section provides a brief overview of some of these.

Continuous learning

Software development is a continuous learning process with the additional challenge of development teams and end product sizes. The best approach for improving a software development environment is to amplify learning. The accumulation of defects should be prevented by running tests as soon as the code is written. Instead of adding more documentation or detailed planning, different ideas could be tried by writing code and building. The process of user requirements gathering could be simplified by presenting screens to the end-users and getting their input.

The learning process is sped up by usage of short iteration cycles – each one coupled with refactoring and integration testing. Refactoring consists of improving the internal structure of an existing program’s source code, while preserving its external behavior. The noun “refactoring” refers to one particular behaviour-preserving transformation, such as “Extract Method” or “Introduce Parameter”. Refactoring does “not” mean: rewriting code, fixing bugs or improve observable aspects of software such as its interface

Refactoring in the absence of safeguards against introducing defects (i.e. violating the “behaviour preserving” condition) is risky. Safeguards include aids to regression testing including automated unit tests or automated acceptance tests, and aids to formal reasoning such as type systems.

Increasing feedback via short feedback sessions with customers helps when determining the current phase of development and adjusting efforts for future improvements. During those short sessions both customer representatives and the development team learn more about the domain problem and figure out possible solutions for further development. Thus the customers better understand their needs, based on the existing result of development efforts, and the developers learn how to better satisfy those needs. Another idea in the communication and learning process with a customer is set-based development – this concentrates on communicating the constraints of the future solution and not the possible solutions, thus promoting the birth of the solution via dialogue with the customer.

Decide as late as possible

As software development is always associated with some uncertainty, better results should be achieved with an options-based approach, delaying decisions as much as possible until they can be made based on facts and not on uncertain assumptions and predictions. The more complex a system is, the more capacity for change should be built into it, thus enabling the delay of important and crucial commitments. The iterative approach promotes this principle – the ability to adapt to changes and correct mistakes, which might be very costly if discovered after the release of the system.

An agile software development approach can move the building of options earlier for customers, thus delaying certain crucial decisions until customers have realized their needs better. This also allows later adaptation to changes and the prevention of costly earlier technology-bounded decisions. This does not mean that no planning should be involved – on the contrary, planning activities should be concentrated on the different options and adapting to the current situation, as well as clarifying confusing situations by establishing patterns for rapid action. Evaluating different options is effective as soon as it is realized that they are not free, but provide the needed flexibility for late decision making.

Deliver as fast as possible

In the era of rapid technology evolution, it is not the biggest that survives, but the fastest. The sooner the end product is delivered without major defects, the sooner feedback can be received, and incorporated into the next iteration. The shorter the iterations, the better the learning and communication within the team. With speed, decisions can be delayed. Speed assures the fulfilling of the customer’s present needs and not what they required yesterday. This gives them the opportunity to delay making up their minds about what they really require until they gain better knowledge. Customers value rapid delivery of a quality product.

The just-in-time production ideology could be applied to software development, recognizing its specific requirements and environment. This is achieved by presenting the needed result and letting the team organize itself and divide the tasks for accomplishing the needed result for a specific iteration. At the beginning, the customer provides the needed input. This could be simply presented in small cards or stories – the developers estimate the time needed for the implementation of each card. Thus the work organization changes into self-pulling system – each morning during a stand-up meeting, each member of the team reviews what has been done yesterday, what is to be done today and tomorrow, and prompts for any inputs needed from colleagues or the customer. This requires transparency of the process, which is also beneficial for team communication.

Another key idea in Toyota’s Product Development System is set-based design. If a new brake system is needed for a car, for example, three teams may design solutions to the same problem. Each team learns about the problem space and designs a potential solution. As a solution is deemed unreasonable, it is cut. At the end of a period, the surviving designs are compared and one is chosen, perhaps with some modifications based on learning from the others – a great example of deferring commitment until the last possible moment. Software decisions could also benefit from this practice to minimize the risk brought on by big up-front design

Empower the team

There has been a traditional belief in most businesses about the decision-making in the organization – the managers tell the workers how to do their own job. In a “Work-Out technique”, the roles are turned – the managers are taught how to listen to the developers, so they can explain better what actions might be taken, as well as provide suggestions for improvements. The lean approach favors the aphorism “find good people and let them do their own job,” encouraging progress, catching errors, and removing impediments, but not micro-managing.

Another mistaken belief has been the consideration of people as resources. People might be resources from the point of view of a statistical data sheet, but in software development, as well as any organizational business, people do need something more than just the list of tasks and the assurance that they will not be disturbed during the completion of the tasks. People need motivation and a higher purpose to work for – purpose within the reachable reality, with the assurance that the team might choose its own commitments. The developers should be given access to the customer; the team leader should provide support and help in difficult situations, as well as ensure that skepticism does not ruin the team’s spirit.

Build integrity in

The customer needs to have an overall experience of the System – this is the so-called perceived integrity: how it is being advertised, delivered, deployed, accessed, how intuitive its use is, price and how well it solves problems.

Conceptual integrity means that the system’s separate components work well together as a whole with balance between flexibility, maintainability, efficiency, and responsiveness. This could be achieved by understanding the problem domain and solving it at the same time, not sequentially. The needed information is received in small batch pieces – not in one vast chunk with preferable face-to-face communication and not any written documentation. The information flow should be constant in both directions – from customer to developers and back, thus avoiding the large stressful amount of information after long development in isolation.

One of the healthy ways towards integral architecture is refactoring. As more features are added to the original code base, the harder it becomes to add further improvements. Refactoring is about keeping simplicity, clarity, minimum amount of features in the code. Repetitions in the code are signs for bad code designs and should be avoided. The complete and automated building process should be accompanied by a complete and automated suite of developer and customer tests, having the same versioning, synchronization and semantics as the current state of the System. At the end the integrity should be verified with thorough testing, thus ensuring the System does what the customer expects it to. Automated tests are also considered part of the production process, and therefore if they do not add value they should be considered waste. Automated testing should not be a goal, but rather a means to an end, specifically the reduction of defects.

See the whole

Software systems nowadays are not simply the sum of their parts, but also the product of their interactions. Defects in software tend to accumulate during the development process – by decomposing the big tasks into smaller tasks, and by standardizing different stages of development, the root causes of defects should be found and eliminated. The larger the system, the more organizations that are involved in its development and the more parts are developed by different teams, the greater the importance of having well defined relationships between different vendors, in order to produce a system with smoothly interacting components. During a longer period of development, a stronger subcontractor network is far more beneficial than short-term profit optimizing, which does not enable win-win relationships.

Lean thinking has to be understood well by all members of a project, before implementing in a concrete, real-life situation. “Think big, act small, fail fast; learn rapidly” – these slogans summarize the importance of understanding the field and the suitability of implementing lean principles along the whole software development process. Only when all of the lean principles are implemented together, combined with strong “common sense” with respect to the working environment, is there a basis for success in software development.

Model Storming: 
Agile Modeling’s practices of light weight, initial requirements envisioning followed by iteration modeling and just-in-time (JIT) model storming work because they reflect deferment of commitment regarding what needs to be built until it’s actually needed, and the practices help eliminate waste because you’re only modeling what needs to be built.
Agility by Self Organization :
It is possible to deliver high-quality systems quickly. By limiting the work of a team to its capacity, which is reflected by the team’s velocity (this is the number of “points” of functionality which a team delivers each iteration), you can establish a reliable and repeatable flow of work. An effective organization doesn’t demand teams do more than they are capable of, but instead asks them to self-organize and determine what they can accomplish. Constraining these teams to delivering potentially shippable solutions on a regular basis motivates them to stay focused on continuously adding value.

Cumulative Flow Diagrams

Cumulative Flow Diagrams have been a standard part of reporting in Team Foundation Server since 2005. Cumulative flow diagrams plot an area graph of cumulative work items in each state of a workflow. They are rich in information and can be used to derive the mean cycle time between steps in a process as well as the throughput rate (or “velocity”). Different software development lifecycle processes produce different visual signatures on cumulative flow diagrams. Practitioners can learn to recognize patterns of dysfunction in the process displayed in the area graph. A truly Lean process will show evenly distributed areas of color, smoothly rising at a steady pace. The picture will appear smooth without jagged steps or visible blocks of color.

In their most basic form, cumulative flow diagrams are used to visualize the quantity of work-in-progress at any given step in the work item lifecycle. This can be used to detect bottlenecks and observe the effects of “mura” (variability in flow).

Visual Controls

In addition to the use of cumulative flow diagrams, Lean Software Development teams use physical boards, or projections of electronic visualization systems, to visualize work and observe its flow. Such visualizations help team members observe work-in-progress accumulating and enable them to see bottlenecks and the effects of “mura.” Visual controls also enable team members to self-organize to pick work and collaborate together without planning or specific management direction or intervention. These visual controls are often referred to as “card walls” or sometimes (incorrectly) as “kanban boards.”

Virtual Kanban Systems

A kanban system is a practice adopted from Lean manufacturing. It uses a system of physical cards to limit the quantity of work-in-progress at any given stage in the workflow. Such work-in-progress limited systems create a “pull” where new work is started only when there are free kanban indicating that new work can be “pulled” into a particular state and work can progress on it.

In Lean Software Development, the kanban are virtual and often tracked by setting a maximum number for a given step in the workflow of a work item type. In some implementations, electronic systems keep track of the virtual kanban and provide a signal when new work can be started. The signal can be visual or in the form of an alert such as an email.

Virtual kanban systems are often combined with visual controls to provide a visual virtual kanban system representing the workflow of one or several work item types. Such systems are often referred to as “kanban boards” or “electronic kanban systems.” A visual virtual kanban system is available as a plug-in for Team Foundation Server, called Visual WIP[20]. This project was developed as open source by Hakan Forss in Sweden.

Small Batch Sizes / Single-piece Flow

Lean Software Development requires that work is either undertaken in small batches, often referred to as “iterations” or “increments,” or that work items flow independently, referred to as “single-piece flow.” Single-piece flow requires a sophisticated configuration management strategy to enable completed work to be delivered while incomplete work is not released accidentally. This is typically achieved using branching strategies in the version control system. A small batch of work would typically be considered a batch that can be undertaken by a small team of 8 people or less in under 2 weeks.

Small batches and single-piece flow require frequent interaction with business owners to replenish the backlog or queue or work. They also require a capability to release frequently. To enable frequent interaction with business people and frequent delivery, it is necessary to shrink the transaction and coordination costs of both activities. A common way to achieve this is the use of automation.

Automation

Lean Software Development expects a high level of automation to economically enable single-piece flow and to encourage high quality and the reduction of failure demand. The use of automated testing, automated deployment, and software factories to automate the deployment of design patterns and creation of repetitive low variability sections of source code will all be commonplace in Lean Software Development processes.

Kaizen Events

In Lean literature, the term kaizen means “continuous improvement” and a kaizen event is the act of making a change to a process or tool that hopefully results in an improvement.

The Lean concept of Kaizen also has a strong influence on the way Agile is being practiced, filling a gap relating to continuous improvement

Lean Software Development processes use several different activities to generate kaizen events. These are listed here. Each of these activities is designed to stimulate a conversation about problems that adversely affect capability and, consequently, ability to deliver against demand. The essence of kaizen in knowledge work is that we must provoke conversations about problems across groups of people from different teams and with different skills.

The evolution of Agile is primarily focused on evolving the product toward a better fit with requirements. In Agile, both the product and the requirements are refined as more is known through experience. Kaizen, a continuous improvement method used in Lean, focuses on the development process itself. When Kaizen is practiced in an Agile project, the participants not only suggest ways to improve the fit between the product and the requirements but also offer ways to improve the process being used, something usually not emphasized in Agile methods. Eckfeldt described the use of Kaizen snakes and project thermometers to capture process improvement feedback.

Daily standup meetings

Teams of software developers, often up to 50, typically meet in front of a visual control system such as a whiteboard displaying a visualization of their work-in-progress. They discuss the dynamics of flow and factors affecting the flow of work. Particular focus is made to externally blocked work and work delayed due to bugs. Problems with the process often become evident over a series of standup meetings. The result is that a smaller group may remain after the meeting to discuss the problem and propose a solution or process change. A kaizen event will follow. These spontaneous meetings are often referred to as spontaneous quality circles in older literature. Such spontaneous meetings are at the heart of a truly kaizen culture. Managers will encourage the emergence of kaizen events after daily standup meetings in order to drive adoption of Lean within their organization.

Retrospectives

Project teams may schedule regular meetings to reflect on recent performance. These are often done after specific project deliverables are complete or after time-boxed increments of development known as iterations or sprints in Agile software development.

Retrospectives typically use an anecdotal approach to reflection by asking questions like “what went well?”, “what would we do differently?”, and “what should we stop doing?”

Retrospectives typically produce a backlog of suggestions for kaizen events. The team may then prioritize some of these for implementation.

A retrospective is intended to reveal facts or feelings which have measurable effects on the team’s performance, and to construct ideas for improvement based on these observations. It will not be useful if it devolves into a verbal joust, or a whining session.

On the other hand, an effective retrospective requires that each participant feel comfortable speaking up. The facilitator is responsible for creating the conditions of mutual trust; this may require taking into accounts such factors as hierarchical relationships, the presence of a manager for instance may inhibit discussion of performance issues.
Being an all-hands meeting, a retrospective comes at a significant cost in person-hours. Poor execution, either from the usual causes of bad meetings (lack of preparation, tardiness, inattention) or from causes specific to this format (lack of trust and safety, taboo topics), will result in the practice being discredited, even though a vast majority of the Agile community views it as valuable.
An effective retrospective will normally result in decisions, leading to action items; it’s a mistake to have too few (there is always room for improvement) or too many (it would be impractical to address “all” issues in the next iteration). One or two improvement ideas per iteration retrospective may well be enough.
Identical issues coming up at each retrospective, without measurable improvement over time, may signal that the retrospective has become an empty ritual

Operations Reviews

An operations review is typically larger than a retrospective and includes representatives from a whole value stream. It is common for as many as 12 departments to present objective, quantitative data that show the demand they received and reflect their capability to deliver against the demand. Operations reviews are typically held monthly. The key differences between an operations review and a retrospective is that operations reviews span a wider set of functions, typically span a portfolio of projects and other initiatives, and use objective, quantitative data. Retrospectives, in comparison, tend to be scoped to a single project; involve just a few teams such as analysis, development, and test; and are generally anecdotal in nature.

An operations review will provoke discussions about the dynamics affecting performance between teams. Perhaps one team generates failure demand that is processed by another team? Perhaps that failure demand is disruptive and causes the second team to miss their commitments and fail to deliver against expectations? An operations review provides an opportunity to discuss such issues and propose changes. Operations reviews typically produce a small backlog of potential kaizen events that can be prioritized and scheduled for future implementation.

There is no such thing as a single Lean Software Development process. A process could be said to be Lean if it is clearly aligned with the values and principles of Lean Software Development. Lean Software Development does not prescribe any practices, but some activities have become common. Lean organizations seek to encourage kaizen through visualization of workflow and work-in-progress and through an understanding of the dynamics of flow and the factors (such as bottlenecks, non-instant availability, variability, and waste) that affect it. Process improvements are suggested and justified as ways to reduce sources of variability, eliminate waste, improve flow, or improve value delivery or risk management. As such, Lean Software Development processes will always be evolving and uniquely tailored to the organization within which they evolve. It will not be natural to simply copy a process definition from one organization to another and expect it to work in a different context. It will also be unlikely that returning to an organization after a few weeks or months to find the process in use to be the same as was observed earlier. It will always be evolving.

The organization using a Lean software development process could be said to be Lean if it exhibited only small amounts of waste in all three forms (“mura,” “muri,” and “muda”) and could be shown to be optimizing the delivery of value through effective management of risk. The pursuit of perfection in Lean is always a journey. There is no destination. True Lean organizations are always seeking further improvement.

Lean Software Development is still an emerging field, and we can expect it to continue to evolve over the next decade.

Lean software development at BBC WorldWide

The lean ideas behind the Toyota production system can be applied to software project
management. This paragraph explains the  investigation of the performance of a nine-person software development team employed by BBC Worldwide based in London. The data collected in 2009 involved direct observations of the development team, the kanban boards, the daily stand-up meetings, semistructured interviews with a wide variety of staff, and statistical analysis. The evidence shows that over the 12-month period, lead time to deliver software improved by 37%, consistency of delivery rose by 47%, and defects reported by customers fell 24%. The significance of this work is showing that the use of lean methods including visual management, team-based problem solving, smaller batch sizes, and statistical process control can improve software development. It also summarizes key differences between agile and lean approaches to software development.
The conclusion is that the performance of the software development team was improved by adopting a lean approach. The faster delivery with a focus on creating the highest value to the customer also reduced both technical and market risks.

Lean software development at IMVU INc

IMVU Inc. (www.imvu.com) is a virtual company where users meet as personalized avatars in 3D digital rooms. Founded in 2004, IMVU has 25 million  registered users, 100,000 registered developers and  reached $1 million in monthly revenue. Over 90 percent of IMVU‘s revenue is from the direct sale of  virtual credits (a form of currency) to users who purchase digital products from its 1.8 million item digital  catalog. IMVU has won the 2008 Virtual Worlds  Innovation Award and was also named a Rising Star  in the 2008 Silicon Valley Technology Fast 50 program. IMVU receives funding from top venture investors Menlo Ventures, Allegis Capital and Bridgescale Partners. Its offices are located in Palo Alto,  CA. (http://crunchbase.com/company/imvu)

Software Development at IMVU
IMVU‘s founders had previously founded  There.com—a virtual world startup that took three  years to build, burned through a ton of money, and  was an abysmal failure after launch. However, from  an engineering perspective, There.com was an  amazing success, as they built it ahead of schedule,  maintained tight quality standards, and solved multiple difficult technical problems. Still, it wasn‘t a  commercial success—and large amounts of time and  money were wasted. As a result, IMVU‘s founding  team decided to build the minimum viable product  and then test it with users—even if the product  seemed only half-built (an engineer‘s nightmare).
As a result, IMVU was one of the startups that pioneered the ―build-just-a-little-and-get-customer-feedback model. This model was only possible because  of the application of several lean principles at the  technical level in the development process.
Lean Principle #1: Specify Value in the Eyes of the  Customer
From the beginning, IMVU‘s founders decided they  wanted to build a culture of ―ship, ship, ship. From a  business perspective, this makes a lot of sense, but  from an engineering perspective, it‘s like pulling fingernails with a pair of rusty pliers:―Bugs were all over the place, extremely ugly looking, and only the most rudimentary features.
In essence, releasing a sub-par product allowed  IMVU to avoid over-production wastes by putting  man hours only into features their customers liked.
Lean Principle #2: Identify Value Stream and  Eliminate Waste
The IMVU team worked hard to cultivate the ―ship,  ship, ship mentality. For example, on their very first  day, most developers were expected to write some  code and push it into production. Even though it was  generally just a small bug fix or a miniscule feature,  this ―release-code-on-the-first-day-of-work idea  seemed revolutionary for most new hires.
Continuous deployment reduced the wastes of overproduction, waiting, and processing. In a traditional  development process, multiple engineers are busy  building multiple features based on the last bit of  stable code. When they try to deploy their feature  after two weeks of work, they find that someone else  deployed a different feature the previous day and the
two features don‘t play well together. Continuous  deployment allows engineers to upload their work  instantaneously – thus ensuring engineers are always  working from the same base code. This avoids  spending extra weeks making the feature code  compatible.
Lean Principle #3: Make Value Flow at the Pull of the Customer
IMVU projects have an eight week Return on Investment (ROI) target. Whenever someone suggests a  small project, they are asked to provide a general  roadmap showing that the project could repay the  time investment in eight weeks. Projects are continuously tested on small numbers of IMVU users –who often had no idea they were part of a bucket test.  If a project shows success, they keep working on it.
After a few weeks, if the numbers shows the project  had zero chance of positive ROI, it is shut down immediately. Over time, as IMVU matures, this project  ROI target is expanded:
Lean Principle #4: Involve and Empower
Employees IMVU implemented the 5 Why‘s process, also known as ―Root cause analysis, to involve and empower its employees during trouble-shooting  processes. 5 Why‘s process is the technique of asking  why five times to get to the root cause of a problem  when it occurs.
in blog posts by Ries, each IMVU engineer has his/her own sandbox which mimicked
production as close as possible. IMVU has a comprehensive set of unit, acceptance, functional, and performance tests, and practiced Test-Drive-Development across the whole team. Engineers build a series of test tags, and quickly run a subset of tests in their sandboxes. Revisions are required if a test fails. To keep developers on the same code before it passed the various tests, IMVU created the equivalent of a Kanban system plus an Andon cord (automated testing and immediate rollback system). Developers are assigned a single task, and not allowed to move onto the next task until their code not only successfully passes the automated testing, but also has successfully deployed. Only then, they can pull the next task from the queue. This means that developers have a little bit of idle time while the tests were running. It also means that code is fully completed before a developer moves on. As a result, engineering is optimized for productivity rather than activity:

Lean Principle #5: Continuously Improve in Pursuit of Perfection
The problem with all this emphasis on ―ship, ship, ship, was different bugs in the code kept taking the site down. Sometimes it was simply a scaling issue— new upgrades worked fine on an engineer‘s computer, but crashed when hundreds of thousands of users tried it. Other times, it was a new employee releasing some feature without understanding how the previous code base worked. From a business perspective, it didn‘t matter what the problem was; if the site was down, IMVU was losing money.
From a technical perspective, each new problem required a different solution. Solving scaling issues is very different than solving a single infinite loop problem. The only practical fix was either cease continuous deployment or institute automated tests that checked the code, plus allowed for immediate code rollbacks if any server started to crash.

Eventually, IMVU architected a series of automated tests that looked at every new code check-in, tested it, and then pushed it onto the live servers. If at any point the code crashed—either during testing or once it started running in the wild—the automated tests instituted a rollback to the last verified good version, and sent a nice little e-mail back to the engineer that said ―We‘re sorry, but it looks like your code ABC caused a problem at XYZ. Afraid we can‘t let your code go live until this is fixed. As a result, the automated testing caught an amazing number of errors, and IMVU management started pursuing massively high quality expectations.

IMVU Successfully implemented lean principles at the technical level in the software development  process. They encountered many common challenges that software companies face: choosing the  right product feature, long development cycle, endless testing and debugging. IMVU found solutions  by sticking with the basic lean principles. They were  able to identify and reduce common wastes in software development process, specifically, overproduction, waiting, process, and defects.

IMVU clearly demonstrated the importance of lean  implementation in the software development process.
The implementation of lean principles cannot turn  software development into a production line environment, with scientific methods for each step of the  way. However, it can help turn a chaotic, constantly  changing process, into a much more predictable, fast  moving, and streamlined process. Lean implementation coupled with brilliant designs and fully engaged  intellectual team can help deliver great software  products.
It would seem that the rapid release cycles called for  by lean principles can only be effective if there is a  comprehensive and rigorous testing environment. An  interesting question is whether IMVU‘s practices  (such as daily release online) would be applicable to  software companies that focus on packaged rather  than online products. In this case, the ―customer is a  combination of the other developers and the ultimate  consumer. IMVU‘s experience challenges the conventional wisdom in software development. Can it be  beneficial to all software companies striving to deliver the right product, at the right time, and at the  right price? Middleton and Sutton  believe that the benefits work across different types of software. Yet,  they also recognize that lean software is far too early  in its evolution.

Lean Beyond Agile

In recent years, Lean Software Development has really emerged as its own discipline related to, but not specifically a subset of the Agile movement. This evolution started with the synthesis of ideas from Lean Product Development and the work of Donald G. Reinertsen and ideas emerging from the non-Agile world of large scale system engineering and the writing of James Sutton and Peter Middleton. David J. Anderson also synthesized the work of Eli Goldratt and W. Edwards Deming and developed a focus on flow rather than waste reduction . At the behest of Reinertsen around 2005,  David J. Anderson introduced the use of kanban systems that limit work-in-progress and “pull” new work only when the system is ready to process it. Alan Shalloway added his thoughts on Lean software development in his 2009 book on the topic. Since 2007, the emergence of Lean as a new force in the progress of the software development profession has been focused on improving flow, managing risk, and improving (management) decision making. Kanban has become a major enabler for Lean initiatives in IT-related work. It appears that a focus on flow, rather than a focus on waste elimination, is proving a better catalyst for continuous improvement within knowledge work activities such as software development.

PEARL XVI : SCRUMBAN – A tool for software production and Support

PEARL XVI : Scrumban is a combination between Scrum and Kanban methodologies for increased applicability and versatility tool for software production and support focused companies.

Scrum is an iterative and prescriptive process for building software using the Agile methodology. A development team plans and commits to completing a certain amount of work in a certain time period called a sprint. At the end of the sprint, the team reviews the work with a product owner. They then hold a retrospective to analyze their processes during the sprint, and determine what can be improved next time.

Scrum can be an effective tool for introducing teams to Agile. Its more rigid structure provides a framework of understanding that is far easier to grasp than the loose nature of Kanban might be for teams used to rigidly planned waterfall-style projects.

The daily standup, planning, review, and retrospective meetings are excellent touch points for periodic checking in with the work that is happening and reviewing with stakeholders. The retrospective itself is the crown jewel of the Scrum framework, and is what enables teams to focus on continuous improvement, or Kaizen.

One of the primary motivations for moving a team to Scrum is to get away from the often restrictive and inefficient processes of the waterfall model. In this model, there is frequently too much time spent on planning and designing before any work begins. After that, teams find themselves unable to respond to change later in the project as the developing and testing proceeds. All of this is why waterfall is a sub-optimal process for software development.

But look at the makeup of a typical Scrum sprint:

  1. Team plans the work that will be worked on over the next sprint.
  2. During planning, teams try to design as many features as possible, so that they can more accurately estimate what they can complete during the Sprint.
  3. During the Sprint, the team develops and then tests their user stories.
  4. At the end of the Sprint, the Product Owner reviews the work completed, and decides which of the stories are shippable and ready for production.

What does this mean? Scrum is a series of miniature waterfall projects wrapped up into iterations called “Sprints.” This is a push-based system, where another solution would be to use a pull-based one. Enter Scrumban.

Scrumban can give teams the power to adapt and change to stakeholder and production needs, without feeling overburdened by their project methodology. It is designed to remove metrics that encourage undesired outcomes. It can restore working time to the team, and avoids unnecessary meetings. And most importantly, it can limit the team’s work in progress so that they can finish what they start to a high standard. Scrumban can remove overhead stress for the development team, increase efficiency, and increase the overall satisfaction for the customer.

Scrumban combines the best features of both methods. It compounds the prescriptive nature of Scrum and the process improvement of Kanban, allowing teams to become Agile and to continually improve their process. Scrumban is becoming especially popular in services industries, where project development and maintenance goes together.

Scrumban is a process for evolving a Scrum instance to be more Lean oriented, supplemented with core Kanban practices such as: visualization, work-in-process limits, work flow management, and making policies explicit. The basic Scrumban implementation principles include:

  • Start with all the ceremonies, artifacts and roles you use now.
  • Agree to pursue improvement towards a more effective process.
  • Respect current roles, responsibilities and job titles.

Scrumban as a combination of a Kanban and Scrum method

Scrum-ban is a software production model based on Scrum and Kanban. Scrum-ban is especially suited for maintenance projects or (system) projects with frequent and unexpected work items or programming errors. In such cases the time-limited sprints of the Scrum model are of no appreciable use, but Scrum’s daily meetings and other practices can be applied, depending on the team and the situation at hand. Visualization of the work stages and limitations for simultaneous unfinished work and defects are familiar from the Kanban model. Using these methods, the team’s workflow is directed in a way that allows for minimum completion time for each work item or programming error, and on the other hand ensures each team member is constantly employed.

To illustrate each stage of work, teams working in the same space often use post-it notes or a large whiteboard. In the case of decentralized teams, stage-illustration software such as Assembla, ScrumWorks, TargetProcess, Rational Team Concert, Eylean board, OnTime, Kanban Tool or JIRA in combination with Jira Agile can be used to visualize each team’s product backlog items, defects and tasks divided into separate phases.

In their simplest, the tasks are categorized into the work stages:

  • Unstarted
  • Ongoing
  • Completed

If desired, though, the teams can add more stages of work (such as “defined”, “designed”, “tested” or “delivered”). These additional phases can be of assistance if a certain part of the work becomes a bottleneck and the limiting values of the unfinished work cannot be raised. A more specific task division also makes it possible for employees to specialize in a certain phase of work.

There are no set limiting values for unfinished work. Instead, each team has to define them individually by trial and error; a value too small results in workers standing idle for lack of work, whereas values too high tend to accumulate large amounts of unfinished work, which in turn hinders completion times. A rule of thumb worth bearing in mind is that no team member should have more than two simultaneous selected tasks, and that on the other hand not all team members should have two tasks simultaneously.

The major differences between Scrum and Kanban are derived from the fact that, in Scrum, work is divided into sprints that last a certain amount of time, whereas in Kanban the workflow is continuous. This is visible in work stage tables, which in Scrum are emptied after each sprint. In Kanban all tasks are marked on the same table. Scrum focuses on teams with multifaceted know-how, whereas Kanban makes specialized, functional teams possible.

In Kanban, Workitems deemed  high priority and needed as soon as  possible have a way of being addressed  ahead of current work. Periodic reviews  of process, tools, team performance, and
WIP limits to find improvements and  eliminate inefficiencies or waste.

When to use Scrumban?

Are you considering implementing Scrumban in your team or business? You should take it into account if you are doing one of these:

  • Projects maintenance,
  • Event-driven work (support, hardening),
  • Problematic projects management (projects with unexpected user stories and bugs),
  • New product development (work preceding sprint development or following sprint development),
  • Continuous management improvement.

Scrumban board gives an easy view of process workflow. It informs about the number of items the team is currently working on as well as already finished ones. It influences the improvement of accountability, responsibility, communication and performance results.

 The example of a Scrumban board

Scrumban is a pull-based system, where the team no longer plans out the work that is committed to during the planning meeting, and instead continually grooms the backlog. The same Scrum meetings (planning, review, and retrospective) can and should still take place, but the cadence of them can be more context-driven. The real key to moving to Scrumban, though, is ensuring that work in progress (WIP) is still limited.

Work-in-progress limits, not Sprints. With Scrum, the amount of work that is ongoing is limited by the Sprint time commitment. But in Scrumban, with no specific time commitment, the team must limit itself through the use of WIP limits on columns within their task board. The goal is always to move tickets in a flow from left to right on the board. If too many issues are in progress, the team is at risk of not finishing anything to high quality standards. Instead, there should be a maximum number of tickets allowed per column. If the number of tickets in that column ever exceeds the maximum, the entire team should swarm onto that column and help move tickets on. This should happen no matter what functional role a team member fills.

Key Scrumban Terms
• Work Backlog – Equivalent to Scrum’s Product Backlog with minor differences for how  Workitems are tracked.
• Workitem– The individual descriptions of functionality needed by the business that either define enhancements or defects.
• Workline – The series of stages in the software development process that pull work from the  Work Backlog and produce shippable business value.
• Stages – The common and discrete steps required to deliver software within an organization.

The Weekly Timeblock (WTB)

The Weekly Timeblock is a reoccurring meeting used for multiple purposes. It should be set up in the  middle of the week (Wednesday are best) so as to avoid holidays and vacation days that typically  occur at the beginning and end of each workweek.
Two to four hours is recommended and set to occur in the afternoon so any preparation work can be  completed ahead of time. The team will use the time they need and leave the rest. Having a projector,  screen, and ample whiteboard space will be conducive for any technical discussions and problem  solving.
The first use of the Weekly Timeblock is for demonstrating any completed work. This is the time  where Developers can show the Product Owner what was completed and get Acceptance. Completed  work can be demonstrated outside of the meeting anytime but it is good for the team to have  every one witness the conclusion of work.
Once Workitems have been demonstrated, the team should reflect on how their development process  is going, any tools they use, and review the data on how Workitems are flowing through the  Workline. They should have frank discussions about any new innovation that should be reinforced or  impediments to be overcome. Clear action items and their owners should be identified along with a  target resolution date.
After the review of recent activity, the team should have a preview of upcoming Workitems the  Product Owner is preparing. The Developers should seek to understand the intent of each item,  asking questions and getting clarity so everyone is familiar with what’s needed. The Developers  should also provide feedback on Workitem scope, structure, and the optimal execution order so the  Product Owner can make any needed adjustments to content and priority.

Standup meetings. The Daily Standup meeting is a concept borrowed from Scrum and serves the same purpose; to coordinate activities and identify impediments. A daily reoccurring meeting should be set up for 30  minutes where the first half is focused on the team members updating everyone on what they did  since the last meeting, what they intend to do before the next meeting, and any impediments that are  blocking, or threaten to block progress on their work. The second half is for any impromptu  discussions necessary to address questions or plan activities. The meeting should be held near the Board so discussions focus around work in progress and updates can be made publically. Ideally the  meeting should occur early in the day, just after everyone has had a chance to get to work, reorient themselves, and think about what they’ll focus on.

In practice, this boils down to redundant statuses that recount information available on the team’s task board. For Scrumban, a more effective method is to refocus on the flow of tickets on the board. That same pattern of yesterday/today/blocked can be transferred to the tickets themselves—the group moves through each column and briefly discusses each ticket and what is necessary to move that ticket rightward on the board. This provides far more context to the team and informs everyone of any major architectural or design decisions.

Metrics. Metrics can certainly be useful, but they are often abused by managers and business stakeholders who want to unnaturally simplify a complex process into a one-dimensional number. Velocity, the amount of story points a Scrum team completes in a single Sprint, is such a metric that incentivizes lower quality at the end of a Sprint as a team scrambles to finish every last story they committed to. When the number fluctuates, as is common with a newer team, the stakeholders begin to question the outputs of the team, and even the effectiveness of Agile itself.

Instead of velocity, a useful Scrumban metric is cycle time. This is the length of time a ticket takes to complete, measured from when it is first began. Over time, a statistical analysis of all tickets in the project can yield a mean cycle time and standard deviation. This can be a useful planning tool at a macro level, as it is trivial to add up the number of stories and multiply by mean cycle time.

The Work Backlog under Scrumban is basically the same as with Scrum. It is a collection of User  Stories and Acceptance Criteria written and maintained by the Product Owner. These individual Workitems describe the business value desired.

To keep the collection organized, additional information is associated with each item:
• ID–A unique identifier (e.g. B#042) for easy searches and reference.
• Project –A label to group a set of work items together according to a theme or purpose.
• Title –A concise, action-oriented description of the business value desired.
• User Story –A formulaic description of who will benefit from the item, what exactly is  wanted, and why it is wanted. The User Story provides the context behind the request and  enables the Developers to understand the goal, and not just take orders.
• Acceptance Criteria –A list of the specific conditions required for the Workitem to be  considered complete.
• Status –An indicator of the readiness of the Workitem, when it is being worked on, and the  final disposition.

When the progress of a Work item through the Workline is blocked

As a team is working on items, it will be fairly common that progress from one stage to another is blocked. These blockages can be internalto the team where the work was more challenging than expected. Whatever the cause the team should “swarm” to help wherever possible to remove the blockage. All team members can assist to one degree or another, even crossing the traditional role boundaries of Coders & Testers to help test, code, prepare environments, research or write documentation.
When a blocking issue is external to the team, such as when a required server hasn’t yet been delivered or subject matter experts are needed for questions, the Scrum Master should attempt to resolve it. The rest of the team can be engaged as needed to help with external issues.
If a Workitem is blocked completely with no resolution in sight, it should be pulled from the Workline and its status changed from ‘In Progress’ to ‘New’ or ‘Draft’ depending on what is needed before the team can pull it back into the Workline again.

Workitem needs to be completed as soon as possible
There are two ways to address this need. First, the Product Owner should be continually maintaining  the order of the Work Backlog such that the items the business considers the most valuable are at or  near the top. It is these top items that the team reviews and makes ‘Ready’ through discussions and  feedback and are pulled into the Workline as capacity opens up.
If there a more urgent need for a work item to be completed than what the standard process  provides, the item’s priority can be changed to expedited and be fast-tracked through to completion.

Defects in Scrumban

Defects are a natural part of any software development effort and dealing with them is straightforward. When defects associated with current Workitems are found they should be addressed before the item is considered complete and ready for demonstration.
Assuming the defect was found during the Testing stage and it is relatively minor, then the item can stay within the Testing stage as the Coder works closely with the Testing to diagnose the issue and provide updated code.
If the diagnosing and fixing effort is much larger however, such as requiring more than a day to complete, then the Workitem should move back to the Coding stage even if the WIP limits are exceeded. The priority for the Coder should be to address the defect and get the Workline flowing again in the right direction.
Sometimes defects are found in areas of the code not currently part of a Workitem in progress. These should be brought to the attention of the Product Owner who can create a new Workitem representing the defect and prioritize it accordingly in the Work Backlog.

Estimation

The statistics kept on how long Workitems take to transit the Workline should be enough to estimate  how long any particular item will take to be completed. For example, if the average number of days is  four, then any new item pulled into the Workline is likely to take four days. By the same logic, if there was a set of Workitems that comprised a release, then the time to complete the entire set is four  times the number of items in the set

This estimation technique works well when the scopes of the Workitems are somewhat consistent.  When they are not, some rough estimates of size, combined with the transit time for those specific  sizes can produce a more accurate estimate although it will take longer to gather enough data to be  confident with the results.Be mindful that any additional time spent by the team should be worth the  effort and if not, abandoned for the simple statistics Kanban provides.

PEARL XII : Kanban : Application of Lean software development to Agile Methodologies

PEARL XII : Kanban : Application of Lean software development to Agile Methodologies.

Most agile m­ethods such as Scrum and XP are already well aligned with lean principles. In 2004, David Anderson pioneered a more direct implementation of Lean Thinking and Theory of Constraints to software development. Under the guidance of experts such as Don Reinertsen, this evolved into what David called a “Kanban system for software development”, and which most people now simply refer to as “Kanban”. ­So while Kanban as applied to software development is new, Kanban as used in Lean Production is over a half century old.

A software development lifecycle process or a project management process could be said to be “lean” if it was observed to be aligned with the values of the Lean Software Development movement and the principles of Lean Software Development.

Bob Charette was invited but unable to attend the 2001 meeting at Snowbird, Utah, where the Manifesto for Agile Software Development was authored. Despite missing this historic meeting, Lean Software Development was considered as one of several Agile approaches to software development. Jim Highsmith dedicated a chapter of his 2002 book to an interview with Bob about the topic. Later, Mary & Tom Poppendieck went on to author a series of 3 books. During the first few years of the 21st Century, Lean principles were used to explain why Agile methods were better. Lean explained that Agile methods contained little “waste” and hence produced a better economic outcome. Lean principles were used as a “permission giver” to adopt Agile methods.

Lean Software Development is an iterative agile methodology originally developed by Mary and Tom Poppendieck. Lean Software Development owes much of its principles and practices to the Lean Enterprise movement, and the practices of companies like Toyota. Lean Software Development focuses the team on delivering Value to the customer, and on the efficiency of the “Value Stream,” the mechanisms that deliver that Value. The main principles of Lean include:

  1. Eliminating Waste
  2. Amplifying Learning
  3. Deciding as Late as Possible
  4. Delivering as Fast as Possible
  5. Empowering the Team
  6. Building Integrity In
  7. Seeing the Whole

Lean eliminates waste through such practices as selecting only the truly valuable features for a system, prioritizing those selected, and delivering them in small batches. It emphasizes the speed and efficiency of development workflow, and relies on rapid and reliable feedback between programmers and customers. Lean uses the idea of work product being “pulled” via customer request. It focuses decision-making authority and ability on individuals and small teams, since research shows this to be faster and more efficient than hierarchical flow of control. Lean also concentrates on the efficiency of the use of team resources, trying to ensure that everyone is productive as much of the time as possible. It concentrates on concurrent work and the fewest possible intra-team workflow dependencies. Lean also strongly recommends that automated unit tests be written at the same time the code is written.

Perhaps the most important Lean activity is to build a value stream map. This means to break down a process into individual steps, and identify which steps add value and which steps do not, thus adding to the waste (muda). The goal, then, is to eliminate the waste and improve the value-added steps (kaizen). An important lean tool helping to manage the work flow is the concept of pull system, which is usually visualized using a Kanban board. In general, we can define the Kanban software process  as a WIP (Work In Process) limited pull system visualized by the Kanban board.

In recent years, Lean Software Development has really emerged as its own discipline related to, but not specifically a subset of the Agile movement. This evolution started with the synthesis of ideas from Lean Product Development and the work of Donald G. Reinertsen and ideas emerging from the non-Agile world of large scale system engineering and the writing of James Sutton and Peter Middleton.  David also synthesized the work of Eli Goldratt and W. Edwards Deming and developed a focus on flow rather than waste reduction . At the behest of Reinertsen around 2005, David introduced the use of kanban systems that limit work-in-progress and “pull” new work only when the system is ready to process it. Alan Shalloway added his thoughts on Lean software development in his 2009 book on the topic. Since 2007, the emergence of Lean as a new force in the progress of the software development profession has been focused on improving flow, managing risk, and improving (management) decision making. Kanban has become a major enabler for Lean initiatives in IT-related work. It appears that a focus on flow, rather than a focus on waste elimination, is proving a better catalyst for continuous improvement within knowledge work activities such as software development.

The Lean Systems Society published its credo at the 2012 Lean Software & Systems Conference. This was based on a set of values published a year earlier. Those values include:

  • Accept the human condition
  • Accept that complexity & uncertainty are natural to knowledge work
  • Work towards a better Economic Outcome
  • While enabling a better Sociological Outcome
  • Seek, embrace & question ideas from a wide range of disciplines
  • A values-based community enhances the speed & depth of positive change

Kanban is an agile methodology for managing the creation of products with an emphasis on continual delivery while not overburdening the development team. Like Scrum, Kanban is a process designed to help teams work together more effectively.

Kanban is a method for managing knowledge work with an emphasis on just-in-time delivery while not overloading the team members. In this approach, the process, from definition of a task to its delivery to the customer, is displayed for participants to see and team members pull work from a queue.

Kanban in the context of software development can mean a visual process management system that tells what to produce, when to produce it, and how much to produce inspired by the Toyota Production System and Lean manufacturing.

The name ‘Kanban’ originates from Japanese and translates roughly as “signboard” or “billboard”. The Kanban method in one of the Kanban methods available today. It was formulated by David J. Anderson is an approach to incremental, evolutionary process and systems change for organizations. It uses a work-in-progress limited pull system as the core mechanism to expose system operation (or process) problems and stimulate collaboration to continuously improve the system. One example of such a pull system is a kanban system, and it is after this popular form of a work-in-progress, limited pull system that the method is named.

In Lean Software Development, the kanban are virtual and often tracked by setting a maximum number for a given step in the workflow of a work item type. In some implementations, electronic systems keep track of the virtual kanban and provide a signal when new work can be started. The signal can be visual or in the form of an alert such as an email.

Virtual kanban systems are often combined with visual controls to provide a visual virtual kanban system representing the workflow of one or several work item types. Such systems are often referred to as “kanban boards” or “electronic kanban systems

Visual control systems are valuable in changing behavior because they display status in an easy-to-see format. This ensures that everyone has a shared understanding of work status and process constraints. Transparency is a key to achieving organizational change

The Kanban method is rooted in four basic principles:

Start with what you do now

The Kanban method does not prescribe a specific set of roles or process steps. The Kanban method starts with the roles and processes you have and stimulates continuous, incremental and evolutionary changes to your system. The Kanban method is a change management method.

Agree to pursue incremental, evolutionary change

The organization (or team) must agree that continuous, incremental and evolutionary change is the way to make system improvements and make them stick. Sweeping changes may seem more effective but have a higher failure rate due to resistance and fear in the organization. The Kanban method encourages continuous small incremental and evolutionary changes to your current system.

Respect the current process, roles, responsibilities and titles

It is likely that the organization currently has some elements that work acceptably and are worth preserving. We must also seek to drive out fear in order to facilitate future change. By agreeing to respect current roles, responsibilities and job titles we eliminate initial fears. This should enable us to gain broader support for our Kanban initiative. Perhaps presenting Kanban against an alternative more sweeping approach that would lead to changes in titles, roles, responsibilities and perhaps the wholesale removal of certain positions will help individuals to realize the benefits.

Leadership at all levels

Acts of leadership at all levels in the organization from individual contributors to senior management should be encouraged.

Kanban Vs Agile

Kanban systems are an approach to scheduling  work. Kanban shares with typical AMs the fact that  requirements are expressed in atomic features (also  known as user stories, work items, Minimum  Marketable Features, or MMF), to be implemented  incrementally. To this purpose, AMs use time-boxed  iterations. – at the beginning of the iteration, the team meets and chooses features from their backlog  that can be done by the end of the iteration. Over the  course of the iteration, they develop those stories,  and at the end of the iteration, they ship them .
Kanban systems are different because they focus on a continuous flow of work, and disregard fixed  iterations. When needed, the team chooses a subset  of features from the backlog and moves them to the  Kanban board. Then, it develops these features one
after the other. Work is delivered as soon as it’s  ready, and the team only works on one – or very few – feature at a time.

Six core practices of Kanban

Anderson identified five core properties that had been observed in each successful implementation of the Kanban method. They were later relabeled as practices and extended with the addition of a sixth.

  1. Visualize
    The workflow of knowledge work is inherently invisible. Visualising the flow of work and making it visible is core to understanding how work proceeds. Without understanding the workflow, making the right changes is harder.
    A common way to visualise the workflow is to use a card wall with cards and columns. The columns on the card wall representing the different states or steps in the workflow.
  2. Limit WIP
    Limiting work-in-process implies that a pull system is implemented on parts or all of the workflow. The pull system will act as one of the main stimuli for continuous, incremental and evolutionary changes to your system.
    The pull system can be implemented as a kanban system, a CONWIP system, a DBR system, or some other variant. The critical elements are that work-in-process at each state in the workflow is limited and that new work is “pulled” into the new information discovery activity when there is available capacity within the local WIP limit.
    WIP limits encourage everyone to work as a team and prevent any one individual from getting too far ahead of anyone else.
  3. Manage flow
    The flow of work through each state in the workflow should be monitored, measured and reported. By actively managing the flow the continuous, incremental and evolutionary changes to the system can be evaluated to have positive or negative effects on the system. Kanban elevates awareness of constraints and forces the team to address them before they can bring additional work into the queue. New work is pulled into the system when there is capacity to handle it, rather than being pushed into the system based on demand. With Kanban, the focus of management and the team becomes the flow of work, not the utilization of the team.
    While Scrum has retrospectives at the end of each iteration to address these items, Kanban explicitly points out constraints in real time and encourages the team to address them as they arise.
  4. Make policies explicit
    Until the mechanism of a process is made explicit, it is often hard or impossible to hold a discussion about improving it. Without an explicit understanding of how things work and how work is actually done, any discussion of problems tends to be emotional, anecdotal and subjective. With an explicit understanding it is possible to move to a more rational, empirical, objective discussion of issues. This is more likely to facilitate consensus around improvement suggestions.
  5. Implement feedback loops
    Collaboration to review flow of work and demand versus capability measures, metrics and indicators coupled with anecdotal narrative explaining notable events is vital to enabling evolutionary change. Organizations that have not implemented the second level of feedback – the operations review – have generally not seen process improvements beyond a localized team level. As a result, they have not realized the full benefits of Kanban observed elsewhere.
  6. Improve collaboratively, evolve experimentally (using models and the scientific method)
    The Kanban method encourages small continuous, incremental and evolutionary changes that stick. When teams have a shared understanding of theories about work, workflow, process, and risk, they are more likely to be able to build a shared comprehension of a problem and suggest improvement actions which can be agreed by consensus.
    The Kanban method suggests that a scientific approach is used to implement continuous, incremental and evolutionary changes.

Common models used are:

  • Theory of constraints (the study of bottlenecks)
  • Deming System of Profound Knowledge (a study of variation and how it affects processes)
  • Lean economic model (based on the concept of elimination of “waste” (or muda, muri and mura)).

Open Kanban

An open source, Agile and Lean based method to deliver value for knowledge work like Information Technology, Software Development, Business, Product Development or Personal organization. On the Lean side it is inspired on the work of Taiichi Ohno (Toyota Production System), Eliyahu Goldratt (Theory of Constraints) and Edward Deming. On the Agile side it takes inspiration from the Agile manifesto signers, and in addition contributions from Alan Shalloway’s Kanban for Teams, Corey Ladas Scrumban and David Anderson’s early Kanban work.

It innovates by making the whole method fully open source and free to improve or modify. Open Kanban was written by Joseph Hurtado, and it has been translated by members of the community to French, Italian, Russian and Ukrainian.

Steps to Get Started

  • Agree on a set of goals for introducing Kanban.
  • Map the value stream.
  • Define some point where you want to control  input. Define what is upstream and who are the upstream stakeholders.
  • Define some exit point beyond which you do  not intend to control.
  • Define a set of work items types based on the work requests that come from the upstream stakeholders.
  • Analyze the demand for each work item type.
  • Meet with the upstream and downstream stakeholders.
  • Create a board/card wall to track the value stream you are controlling.
  • Optionally, create an electronic system to track and report the same.
  • Agree with the team to have a standup meeting in front of the board; invite upstream and downstream stakeholders .
  • Agree to have regular operations review meeting for retrospective analysis of the  process; invite upstream and downstream stakeholders .
  • Train the team on the new board, WIP limits and the pull system. Nothing else in  their world should have changed.

VersionOne Kanban Boards

VersionOne provides a single platform to manage all your projects, from Scrum to Kanban and everything in between, without sacrificing cross-project visibility. Visualize, manage and optimize your workflow using VersionOne’s custom Kanban boards.

With some teams practicing Scrum and others choosing Kanban, it can be challenging for managers to get a clear, integrated view of status across all projects. VersionOne’s Kanban boards make it easy to manage all projects in a single tool without sacrificing cross-project visibility and consolidated reporting. When teams are practicing lean methods within iterations, VersionOne provides the Kanban capabilities to help agile teams visualize and manage their workflow processes.

Agile teams can Create custom Kanban boards using named columns to illustrate each stage of  workflow process, and an expedited track for high priority items. Work items are visually represented using story cards and can be pulled from prioritized queue and moved through the workflow easily using drag-and-drop software functionality.

Managing the Kanban board in VersionOne provides real-time visibility for the entire team with full accessibility for remote members. Agile Teams see at a glance what work items are in progress and where they are in the development process. Visual cues within the Kanban board identify items that are blocked, held up in wait queue, or have exceeded their aging threshold, providing an early indicator of potential problems.

VersionOne software helps maximize  flow of work by setting explicit WIP limits across single or multiple contiguous columns to determine how many items can be worked on at each stage of  process. Visual cues within the Kanban board lets team know when WIP limits are exceeded to trigger conversations and refocus the agile team’s efforts on the right work to ensure a continuous flow.

VersionOne enables to Track and measure the average time to complete a work item using Cycle Cycle Time reporting capabilities. The ability to track cycle time metrics gives visibility into past performance to help  increase the predictability of  delivery and drive continuous process improvement.

The Cycle Time clock starts when work begins on a request and ends when the item is ready for delivery. Cycle time is a mechanical measure of process capability.

VersionOne can be used Identify and resolve bottlenecks using Cumulative Flow reports to see how well items are flowing through the process, Determine where work is stalled and where you should focus improvement efforts.

The Cumulative Flow diagram provides a graphic depiction of how work items are moving through various statuses on the way to being “Done”. It shows the total scope of a project, grouped by status, and thus lets us know how much of that scope is in a particular status at a given time.

As highlightedd, the use of a basic Kanban board (manual and/or virtual for distributed teams)  serves to bring everyone onto the same page, improve focus and prompt questions that lead to revision of policies and practices. When the team’s  experience grows, the Kanban board will change  to expand and refine the workflow representation, thereby reflecting a more sophisticated level