PEARL XVII:
In Agile Software Development Methodology, the Multi Dimensional Testing Coverage Matrix of what needs to be tested should be defined early. The importance of this has grown exponentially the past few years, as the test coverage matrix has become increasingly complex.
Rapid prototyping and development techniques combined with Agile development methodologies are pushing the envelope on the best practice of testing early and testing often. Keeping pace with the quick development turn-around and shorter time to market and being adaptive to late changes in requirements requires effective management of quality process. The use of multidimensional test coverage matrix which maps of test artifacts – test cases, test defects, test fixtures – mapped to the requirements – needs, features, use cases and supplementary requirements – as a QA scheduling and planning tool defined early in the project, though claimed to have been practiced, has been largely overlooked by the software industry.
View of Agile Delivery Life Cycle
User needs in an agile process are defined by a story (sometimes captured as use-cases and features) planned to be implemented iteratively. Work break down for development (in iterations) of these use-cases and features is defined in terms of tasks.
Test coverage can help in monitoring the quality of testing, and assist in directing the test generators to create test cases that cover areas that have not been tested before.
Get Better Structure of the Product: Understanding the end-to-end data flow of an application helps to analyze the root cause of a defect. Split the complete product into small segments that will help team to get a clear structure of a product as well as flows and dependencies of each and every part of modules.
Depending on Windows/Mac/Linux/Unix platforms, different types of browser-based tools are important for agile testers to troubleshoot the defects. For example, for Http(s) tracking tools – Parse Proxy, Http Watch, WebScarab, Http Debugger Pro, Archillies, and log tracing tools- Winscp & Putty, XML SPY,etc. These dependencies shall be clearly depicted in the Multidimensional Test coverage Matrix.
The agile team needs to develop good test scenarios for future automation, end-to-end testing and product lifecycle analysis, to help in organizational process growth
Acceptance TDD (ATDD). Team can do TDD at the requirements level by writing customer test, the equivalent of a function test or acceptance test in the traditional world. Acceptance TDD is often called behavior-driven development (BDD) or story test-driven development, where you first automate a failing story test, then driving the design via TDD until the story test passes (a story test is a customer acceptance test).
Requirement Types
There are several categories of requirements, these are: Stakeholder, HighLevel Business, Application, Data, Security, Test (Quality), Supplementary, and Change
Management. Some of these categories have a lower level of granularity associated with them,
Stakeholder Requirements
These usually come from the CEO of the company, or the body of the Board of Directors. These are very high-level. They usually focus on a major “problem” that they want solved. These can also include high-level expectations for: Performance (“instantaneous”), Availability (“24x7x365”) and Security (“totally secure”).
High-Level Business Requirements
High-level requirements will fulfill Stakeholder Requirements. They are traced directly from them. High-Level business requirements trace to Application Requirements.
Application Requirements
Application requirements drive the development of the application. They fall into the following categories; Use Case Requirements, Business Requirements/Rules, Functional Requirements, Technical Requirements, User Interface Requirements, “Look and Feel” Requirements and Navigation Requirements. There are some very fine lines between these types, and the ultimate strategy may be to combine these categories as needed.
Use Case Requirements
These start in the form of Use-Case Documents, but can be broken down into individual
requirements. The use case describes the basic flow of the system, alternate paths through the system. Business Rules, Navigation and User Interface requirements are all derived from the use case, and are traced from it.
Business Requirements/Rules
A Business Requirement or Rule is something that is specific to the business. These are
sometimes listed as “non-functional” requirements, as they do not have any function to them. The rules exist whether or not a system exists. An example of this would be, for a transportation firm,“Trucks only can travel 55 mph,” and another would be “We accept no bid less than $100,000.” This is also likely a list of data that is required to be captured, with the data including the data items needed, such as Name, address, etc. If there is a business need for data to be in a certain format, or minimum or maximum length for a text field, for example, that too is captured here. Any edit, validation, or default value that serves a business purpose is also in this category. This list of business rules is often captured in a business catalog. These are all should be traced from the use case.
Functional Requirements
These are driven and traced from the Business Requirements and Rules. They describe how a business rule is carried out, or how data is captured. These are usually in the format of “system shall do this or do that.”
Technical Requirements
These are derived during the low level design process, and sometimes during development. They are driven from and traced from the Functional Requirements. They are not “user” features of the system, but describe the low-level required actions of the system, that only a designer or developer would know. For example, “When transaction is being processed, and an error of type XYZ occurs, a rollback of the database transaction is required.” They can also include details of an error message, and error handling requirements.
User Interface Requirements
These are driven from Functional and Use Case Requirements, are traced from them both, depending on where they were derived from. They include items such as screen layout, tab flow, mouse and keyboard use, what controls to use for what functions (e.g. radio button, pulldown list), and other “ease of use” issues.
“Look and Feel” Requirements
These are driven from Functional and Business Requirements, and are traced from them both, depending on where they were derived from. These can be categorized as the “non-functional”
User Interface type requirements. They deal with how fields look, regarding font, font size,
colors, graphics, etc.
Navigation Requirements
These are driven and traced from the Use Case, as the Use Case lists the flow of the system, and the Navigation Requirements depict how that flow will take place. They are usually presented in a storyboard format, and should show the screen flow of each use case, and every alternate flow. Additionally, they should state what happens to the data or transaction for each step, for example, what happens to the data entered on a previous screen if the user decides to “go back?” They include the various ways to get to all screens, and an application screen map should be one of the artifacts derived in this category of requirements.
Based on the above Application Requirements, the Data Requirements are derived, and are traced from the Application Requirements.
Data Requirements
Based on the data needed in the Business Rules, and taking other requirements types into consideration, the database platform, database tables, columns, data types, size and other database attributes can be created.
Security Requirements
These are traced from wherever they need to be traced from, as they encompass all requirements that have do directly with the application . They mainly will come from Business Rules.
Test (Quality) Requirements/Test Cases
The Test Cases are traced from all types of requirements, and can be percived as the all en compassing box that all requirements are inside of. This shows that all requirements shall have test cases traced to it. If that test case fails during testing, the team should be able to see what higher-level requirement would not be fulfilled.
Supplementary Requirements
Traced from Data and Application requirements, these refer to the technology infrastructure and operations type of requirements, such as hardware and network requirements. There are many Operations Requirements that must also be captured, in the categories of, Administration Requirements, Maintenance Requirements, Disaster/Recovery Requirements, Skill Set of personnel Requirements, Change Management Requirements, User Documentation Requirements, Training Requirements, and Demo/Presentation Requirements. Other nonfunctional requirements that do not fall into any other category would fall into this “catch-all” category.
Carefully Define Testing Coverage Matrix: With the help of multi-dimensional matrix, define the coverage requirements for a vital part of the specifications process at an early stage. The size of the coverage matrix has an important influence on staffing needs and QA budgets. Defining the testing coverage matrix early benefits the management to distribute resources and have better control and analysis of what can be done internally versus what should be outsourced.
Test coverage in the test plan states what requirements will be verified during what stages of the product life. Test Coverage is derived from design specifications and other requirements, such as safety standards or regulatory codes, where each requirement or specification of the design ideally will have one or more corresponding means of verification. Test coverage for different product life stages may overlap, but will not necessarily be exactly the same for all stages. For example, some requirements may be verified during Design Verification test, but not repeated during Acceptance test. Test coverage also feeds back into the design process, since the product may have to be designed to allow test access .
In order to reproduce production scenarios or customer-specific issues, development and test teams need to be able to change operating systems, browsers, and databases, or infrastructure parameters such CPU, memory, disk size, and network configuration. These scenarios must be addressed in the multidimensional test coverage matrix.
The use cases descriptions define the main success scenarios of the system. However, not every use case scenario ends in a success for the user. While elaborating the use-cases using the descriptive text to capture these alternate paths, new aspects of the systems come to light when exceptions are encountered (non-happy path behavior of the system is being captured).
Spence and Probasco refer to them as overloading the term requirements, a common point of confusion with Requirements Management. These may not be clear from the user needs and system features captured, but they are a very vital and essential aspect of the system behavior. To ensure that the system meets these requirements and for coverage to be effective, these have to be elicited clearly and traced completely. Alternate paths may also be captured using a usability (scenario) matrix . While the use cases are mapped against features (or cards) which are planned for the iteration, so can the use-cases, the use-case scenarios that stem from these and so on, cascading to the test cases (and test artifacts)
Note that the usage of the application flow, even though captured, could end-up varying the application flow based on the data . Supplementary requirements corresponding to the architectural requirements for the system cannot be mapped unless captured separately. These remain outside the functional requirements modeled by the use cases.
A sample list of business rules that have to be followed could be summarized.
When the mapping of the test case flows across functionality is carried, it becomes evident that the granularity of details falls short; when mapping the coverage of the test flows against the business rules in the tabular representation.
For example, Based on the feature set as set out it is possible that any one of the flows used to ensure coverage of business requirement 1 could as well serve for business requirement 2. However, on closer scrutiny the test case flow that tests non-happy path scenario of business requirement rule 2 requires a further elaboration of the test flows against feature set.
Such gaps and inadequacies will come to light in a multidimensional test coverage matrix that is not granular and consequently, the test coverage falls short. Tracing every non functional, business and non-business requirement to test cases and scenarios should increase the confidence and coverage of testing and QA activities that can be performed. The usability flows and concrete test cases that cover the requirements and needs can be formulated and with each iteration, targeted test cases could be identified to be run or executed to address within the specific build.
Test Coverage matrix is really multi-dimensional and to be effective QA artifacts, they have to transcend the various phases of the development process – initiation, elaboration, construction and transition. Further, it has to be a “living” artifact, one that is updated with each iteration
Multi Dimensional Test coverage matrix example
Within iterations, a set of acceptance and regression tests have to be scheduled and performed to meet the exit criteria. Features and stories (in the form of cards) are planned in iterations in an agile methodology. With multi dimensional test coverage matrix and mapping of the test cases to features, use cases and defects, optimum test planning assuring the software quality within each build/release becomes effortless and convenient. By establishing effective multi dimensional test coverage matrix, helps to answer some of the following questions
1. What test cases should be selected to run for the current build – verify fixed defects, regression suite for the current fixes, apart from the base code smoke and build acceptance tests?
2. What impact does change in a specific set of non-functional and functional requirements have on the QA testing process in arriving at test estimates?
3. How can defects identified be mapped to the requirements that the iteration was scoped to achieve? And what surround testing and re-testing have to be carried out to validate before the defects can be closed out or new but related ones identified?
4. What change requests were brought about by the most recent build or iteration and what does this new change entail?
Testing coverage matrix also establishes tracking back to the exact requirements being implemented in the iteration improving coverage and confience in the quality process. This is of greater significance in agile projects where requirements documentation isn’t complete as requirements continue to evolve with each build or iteration. Agility ensures the process (and the product) is adaptive to changing requirements and using testing coverage for QA activities ensures that verification keeps up with these changes. impact on quality
Multi Dimensional Attributes of Test Coverage Matrix
Multi Dimensional Test coverage matrix defines a method for testing a complex software system, wherein the complex software system has interrelated system programs.
The method comprising the steps of:
- Determining linkages between interrelated system programs using an multidimensional test coverage matrix;
- identifying a change in one or more of the interrelated system programs;
- applying the multidimensional test coverage matrix such that it depicts information pertaining to a selected test scenario and test cases pertaining to the identified change,
- And also to depict other test scenarios and test cases linked by two or more levels through at least one of dependencies and tributaries.
- The test coverage matrix is adapted to depict linkages across three or more levels of dependency;
Multi dimensional Test coverage matrix enables to keep track of the complex relationships between various components of complex software system and to assist in the regression testing needed in the integration and subsequent modification of the complex software system. it is used to store the relationship information between test cases (predetermined scenarios involving a series of specified transactions to test the behavior of one or more software components of the complex software system) and/or the individual components of the complex software system
Dependency Scenario
Pearls of Wisdom 