PEARL XXI : PRISMA: Product Risk Assessment for Agile projects

PEARL XXI : PRISMA: Product  Risk Assessment for  Agile projects

Risk assessment and management is the backbone of sequential development models but how does it fit in agile  environments? How can we be sure to identify new risks  when they emerge and to ensure our understanding of  all risks remains accurate? In agile much emphasis is on  communication. Perfect for development issues where  mistakes can be discussed and fixed. But product risk are  not by nature iterative: they is absolute and exists all the  time and making mistakes in dealing with them may not  acceptable. Hence the discussion and consensus approach  needs to be slightly formalized by the use of a systematic  method and process. That’s where PRISMA comes in

Testing activities can be seen as mitigating product risk. A product risk is defined as a risk which is directly related to a potentially failing product. Risk based testing is an approach for developing and prioritizing tests based upon the impact and likelihood of failure of the functionality to be tested. Likelihood is the chance that the software contains defects (caused by for example poor programming, high complexity, etc.). Impact is an indication of the consequences when the software fails.

PRISMA (Product RISk Management) is an approach for identifying the areas that are most important to test, i.e., identifying the areas that have the highest level of business and/or technical risk. The PRISMA method has been bottom-up developed by Improve Quality Services in practice over a large number of years. PRISMA has been proven to be successful in supporting (test) organizations as they apply risk-based testing.
Today, it is taught at several universities to IT students. The PRISMA approach especially supports the test professional in performing product risk identification and product risk analysis as well as in working in close co-operation with stakeholders.
Product Risk Matrix

PRISMA

The central theme in the PRISMA process is the creation of the so-called product risk matrix. For each product risk identified, the impact of possible defects and the likelihood of these defects occurring is determined. By assigning numeric values to both impact and likelihood, a product risk (test item) can be positioned in the product risk matrix.
The standard risk matrix is divided in four areas each representing a different level and type of risk. A different level and/or type of risk should also imply a different test approach, to be documented in a (master) test plan. The product risk matrix can thus used as a basis for all testing performed in a project.

A picture is often worth more than a thousand words. Presenting risk assessment results in a diagram is usually much more effective than in tabular form with many numbers. The table becomes indecipherable very quickly, and often stakeholders lose themselves in a number based discussion.

Presenting the results of a risk analysis in a matrix format, as in a PRISMA product risk where impact is on the horizontal axis, likelihood is on the vertical axis, and the four quadrants each represent a level and type of risk – generally provides a much better basis for discussing and validating the product risks.
Agile
Since risk mitigation is one of main objectives of Agile, an approach such as PRISMA can fit into an Agile development project perfectly. In practice PRISMA has proven to be a relatively light weight approach (unlike some), focused on producing tangible results, e.g., the product risk matrix and a differentiated risk-based test approach. Most often when organizations come from a more traditional environment using a structured testing approach such as TMap, many testing practices are removed from dayto-day practice.

Test management approach (TMap) is a software testing methodology. TMap is a method which combines insights on how to test and what to manage, as well as techniques for the individual test consultant.

One of the testing practices that is still necessary is a product risk assessment which determines where and how to focus the limited test resources to effectively meet the project deadlines.

Where some methods use very detailed approaches for product risk assessment, PRISMA is generally considered relatively light weight and result-driven. In fact, from Agilists experience, most projects that convert to Agile software development keep PRISMA as one of their core testing practices. Note that in Agile the team is explicitly responsible for the quality of the product.

The risk assessment process
How is PRISMA applied in Agile software development?

Risk based testing with “Risk Poker” in agile projects

One of the most frequently asked questions about testing, both in traditional and Agile projects, is: “How much testing should be done”? In some traditional projects managers may want the team to ‘test everything’. They want to be absolutely sure that the system is completely tested before it is released into the market, to prevent problems – or even claims – in production. However, testing the entire system in every possible way is impossible. No organization is willing to spend sufficient resources for ‘exhaustive testing’ and pressure on budget and release schedule will not allow for the required effort.
James Bach, leading proponent of Exploratory Testing, introduced the concept of ‘good enough testing’ in 1997. This concept is helpful in understanding the risk based testing approach. Agile projects are usually not striving to develop ‘the absolute perfect software’.
The concept of ‘a potentially useful version of working product’ in Scrum actually means that the software is working ‘good enough’ to take it into production.

‘Good enough’ in this context is defined as: providing sufficient benefits, having no critical problems and the benefits of releasing now outweigh both the consequences of non-critical problems and delaying the project for further testing.

Risk Poker is an approach for product risk based testing in agile projects. The process of Risk Poker is similar to the way that Planning Poker is done, e.g. in Scrum, except that it will result in risk identification and risk analysis rather than estimations and story points.
What are the reasons for applying risk based testing with Risk Poker in agile projects – and what are the benefits?
1. Most agile methods and frameworks – like Scrum – are time-boxed. Iterations have a fixed duration, so both development and testing activities are by definition limited to a pre-defined timeframe. Risk based testing provides an excellent answer to the problem ‘how much testing’ by ensuring that the most important testing has been done within the available time. Therefore risk based testing is a very suitable approach for time-boxed development methods.
2. Just like Planning Poker, Risk Poker is a team-based activity and decisions are made by achieving consensus.
3. In the Scrum process, Risk Poker can be easily combined with Planning Poker in the Planning meeting. They complement each other, because information about business value from the Product Owner (PO) will provide input for the impact component of product risk, and the question-and-answer game about product risks will be input for estimating the testing effort in Planning Poker.
4. User stories are very suitable entities to be used as ‘risk items’ in a product risk analysis. In agile projects, risk identification comes down to identifying user stories.
5. Agile is all about ‘working software’, Risk Poker is a light-weight approach to achieve the most appropriate balance between sufficient quality and acceptable risk – within the available constraints in time and resources.

Product risks are derived from documents (i.e., the list of backlog items assigned to the next sprint and user stories) and are typically identified in a brainstorm session(s). Of course the approach largely depends on the Agile approach that is being used and the cycle time. Based on agilists experience, longer sprints of four weeks or one month are most common. The sprint team is often also the PRISMA team performing the product risk assessment.

“External” stakeholders are contacted and asked for their input or actively participate in the process. It is usually carried out as a focused meeting, where the team runs through the PRISMA process as described below. At the end of the meeting the team agrees on the product risk matrix and thus the focus of testing.

Risk poker

Prior to the planning meeting, the team should determine which factors influence the quality of the delivered software. Typical factors for likelihood are complexity, new development (level of re-uses), interrelations (number of interfaces), size, technology and (in-)experience of team. The team itself decides which factors for likelihood are to be taken into account during the Risk Poker.
Concerning the impact, influencing factors can be business importance (i.e. selling item), financial damage, usage intensity, external visibility and legal sanctions. The Product Owner will decide (together with stakeholders) which of these factors for impact are to be taken into account.

Risk Planes

Having the list of product risks, they are now scored (separately for likelihood and impact) using the essentials of the planning poker technique as often practiced in agile projects. Planning Poker is a consensus-based technique for estimating. It is a variation of the Wideband Delphi method.
The PRISMA risk poker is uses the list of product risks (user stories) to be tested and several copies of a deck of cards. The decks have numbered cards and often use the sequence: 0, ½, 1, 2, 3, 5, 8, 13, 20, 40, 100, and optionally a “?” (unsure) and a coffee cup (I need a break). A common variation is not using a deck with numbers but colored cards, e.g., dark green, light green, yellow, orange and red, relating back to the “1 to 5” value set. This is practiced since the meaning of the numbers from the deck often lead to much discussion, and are ambiguous in the PRISMA context, when using them to estimate likelihood and impact.
Each team member receives a deck of cards with varying values (or colors). After a short explanation of the product risk item (user story), the moderator (e.g., a SCRUM Master) calls for an estimate for either likelihood or impact. After a few seconds of contemplation, each team member selects a card, without showing it to the other team members, and at a set time, all show their selected cards. It is important that all cards are shown at once, to prevent ‘peer pressure’ towards a lower or higher number (or color). If the numbers (or colors) are essentially the same, the moderator writes down the median value. If they differ wildly, the lowest estimator and highest estimator briefly explain their choice essentially going back to the PRISMA factors for likelihood and impact. Often then agreement is achieved for a number (or color) based on that discussion. If no agreement
is reached, the moderator, business owner (for impact) or lead developer (for likelihood) act as a tie breaker and chooses a number (or color) from within the range. It is important to move quickly to the next product risk item. Optionally, an egg timer can be used to limit time spent in discussion of each item. One common variation is providing each team member with a limited number of each value or color, and having them ‘use up’ each value card in the process. This prevents the tendency of some people to stick to very
high or very low scores for all product risks.

Protection Poker

Without infinite resources, software development teams must prioritize security fortification efforts to prevent the most damaging attacks. The Protection Poker “game” is
a collaborative means for guiding this prioritization and has the potential to improve software security practices and team software security knowledge.

Playing cards in hand, the software development team members stare silently at their cards. Players glance at each other while pensively considering their options. Grant,
the development manager, announces, “Everybody ready?” and each member lays down a card. At once, the silence erupts into a team-wide conversation of opinions, perspective, and debate.
No, this isn’t your secret lunchtime poker game in the broom closet. Nor is this a naïve “team-building” activity from human resources. The team is playing Protection Poker,1 a new software security “game.”
Protection Poker is an informal game for security risk estimation that leads to proactive security fortification during development and prioritizes security- related validation and verification (V&V). Protection Poker provides structure for collaborative misuse case development and threat modeling that plays off the participants’ diversity of knowledge and perspective. The entire extended development team gets involved—software developers, testers, product managers or business owners, project managers, usability
engineers, security engineers, software security experts, and others. Protection Poker is based on a collaborative effort estimation practice, Planning Poker, which many agile software development teams use. (The “rules” of Planning Poker don’t at all resemble
actual poker’s rules, except that each participant hides his or her cards from the other participants until a designated time. Collocated teams often use special cards to do their estimation that contain only selected values.  The Red Hat IT team utilized the Scrum agile software development methodology and “played” Protection Poker during its biweekly iteration planning meetings over a four-month period.
Protection Poker
Protection Poker is a simple but effective software security game. Its tangible output is a list of each requirement’s relative security risk. The team can use this relative risk to determine the type and intensity of design and V&V effort the development team must include in the iteration for each requirement. The team can then use this list to help prioritize security engineering resources toward software areas with the highest risk of attack based on factors such as how easy the new functionality is to attack and the value
of the data accessed through the functionality. Consequently, the team properly estimates the necessary effort to implement the requirement securely, so it can proactively plan which resources are needed for secure implementation. This prioritization and increased
knowledge should lead a team toward developing more secure software. Protection Poker works best for teams that use an iterative development process with relatively short iterations, as agile software development teams often do

Protection Poker and Planning Poker are Wideband Delphi techniques. (Planning Poker’s creator likely chose its name for the catchy alliteration, whereas the term Wideband Delphi might have seemed less accessible to agile teams.) Wideband Delphi is based on the Delphi practice, developed at the RAND Corporation in the late 1940s for the purpose of making forecasts. With the Delphi practice, participants make estimates individually and anonymously in a preliminary round. They collect, tabulate, and return the first-round results to each participant for a second round, during which they must again make a new forecast regarding the same issue. This time, each participant knows what the other participants forecasted in the first round, but doesn’t know the other participants’ rationale behind those forecasts. The second round typically results in a narrowing of the group’s range in forecasts, pointing to some reasonable middle ground regarding the issue of concern. The original Delphi technique avoided group discussion to enable candid and anonymous input.
Barry Boehm created the Wideband Delphi technique as a variant of the Delphi technique where group discussion occurs between rounds in which participants explain why they’ve chosen their values. Wideband Delphi is useful for coming to some conclusion regarding an issue when the only information available is based more on experience than empirical data.

Inspired by Planning Poker Protection Poker uses the relative measures of ease points and value points in its security risk computation (for example, one requirement is five times easier to attack than another). Team members vote on their estimate of relative measures for ease of attack and asset value.

The team is constrained to nine possible values—for instance, 1, 2, 3, 5, 8, 13, 20, 40, and 100 (which Planning Poker uses)—for ease points and value points.
The game uses these particular values because humans are more accurate at estimating small things; hence more possible small values exist than large ones.5 Additionally, team members can do their estimations more quickly with a limited set of possible values. For
example, why argue over whether a requirement is 40 or 46 times easier to attack than another? At that point, we can only really know that the requirement is “a lot easier” to attack.

Using Protection Poker should reduce vulnerabilities in the product through an overall increase of software security knowledge in the team. We observed four major benefits to using the Protection Poker practice:

Security risk estimate and ranking : Albeit based on relative estimates, Protection Poker quantifies software security risk, which a team can then use to rank requirements. This ranking can help developers plan explicit actions to reduce security risk. The extended team obtains estimates via all members’ expert opinions. Incorporating these opinions leads to improved estimation accuracy,particularly over time.

Adaptation of requirement to reduce security risk. The initial requirement might not reflect the need for security functionality, such as role-based access or logging. Through the extended team’s think-like-an-attacker brainstorming, these needs could surface, and the team can update the requirement accordingly.

Proactive security fortification to reduce security risk. Teams who don’t consider security issues as they develop the software might realize too late that they didn’t allocate enough time in the development schedule to build a secure product, sometimes resorting to shipping a knowingly insecure one. Through Protection Poker, before requirement implementation begins, the extended development team has a chance to brainstorm and decide what explicit actions are necessary to reduce security risk, such as conducting a security inspection or intense input-injection testing for a Web form. The team can plan these explicit actions into the implementation schedule.
Software security knowledge sharing. Protection Poker inspires a structured discussion of security issues that incorporates the extended development team’s diverse perspectives. This discussion improves the team members’ knowledge and awareness of what is required to build security into a product.