PEARL XVIII : Elucidation on ATDD – Acceptance Test Driven Development

PEARL XVIII : Elucidation on ATDD – Acceptance Test Driven Development 

TDD helps software developers produce working, high-quality code that’s maintainable and, most of all, reliable. Customers are rarely, however, interested in buying code. Customers want software that helps them to be more productive, make more money, maintain or improve operational capability, take over a market, and so forth. This is what we need to deliver with our software—functionality to support business function or market needs. Acceptance test-driven development (acceptance TDD) is what helps developers build high-quality software that fulfills the business’s needs as reliably as TDD helps ensure the software’s technical quality.

Acceptance Test Driven Development (ATDD) is a practice in which the whole team collaboratively discusses acceptance criteria, with examples, and then distills them into a set of concrete acceptance tests before development begins. It’s the best way to ensure that we all have the same shared understanding of what it is we’re actually building. It’s also the best way to ensure we have a shared definition of Done. .

Acceptance TDD helps coordinate software projects in a way that helps us deliver exactly what the customer wants when they want it, and that doesn’t let us implement the required functionality only half way.

An essential property of acceptance TDD is that it’s a team activity and a team process.

Acceptance tests are specifications for the desired behavior and functionality of a system. They tell us, for a given user story, how the system handles certain conditions and inputs and with what kinds of outcomes. There are a number of properties that an acceptance test should exhibit; 

An important property of acceptance tests is that they use the language of the domain and the customer instead of geek-speak only the programmer understands. This is the fundamental requirement for having the customer involved in the creation of acceptance tests and helps enormously with the job of validating that the tests are correct and sufficient. Scattering too much technical lingo into our tests makes us more vulnerable to having a requirement bug sneak into a production release—because the customer’s eyes glaze over when reading geek-speak and the developers are drawn to the technology rather than the real issue of specifying the right thing.

By using a domain language in specifying our tests, we are also not unnecessarily tied to the implementation, which is useful since we need to be able to refactor our system effectively. By using domain language, the changes we need to make to our existing tests when refactoring are typically non-existent or at most trivial.

Concise, precise, and unambiguous

Largely for the same reasons we write our acceptance tests using the domain’s own language, we want to keep our tests simple and concise. We write each of our acceptance tests to verify a single aspect or scenario relevant to the user story at hand. We keep our tests uncluttered, easy to understand, and easy to translate to executable tests. The less ambiguity involved, the better we are at avoiding mistakes and the working with our tests.

We might write our stories as simple reminders in the form of a bulleted list, or we might opt to spell them out as complete sentences describing the expected behavior. In either case, the goal is to provide just enough information for us to remember the important things we need to discuss and test for, rather than documenting those details beforehand. Card, conversation, confirmation—these are the three Cs that make up a user story. Those same three Cs could be applied to acceptance tests as well.

Yet another common property of acceptance tests is that they might not be implemented (translation: automated) using the same programming language as the system they are testing. Whether this is the case depends on the technologies involved and on the overall architecture of the system under test. For example, some programming languages are easier to inter-operate with than others. Similarly, it is easy to write acceptance tests for a web application through the HTTP protocol with practically any language we want, but it’s often impossible to run acceptance tests for embedded software written in any language other than that of the system itself.

The main reason for choosing a different programming language for implementing acceptance tests than the one we’re using for our production code (and, often, unit tests) is that the needs of acceptance tests are often radically different from the properties of the programming language we use for implementing our system. To give you an example, a particular real-time system might be feasible to implement only with native C code, whereas it would be rather verbose, slow, and error-prone to express tests for the same real-time system in C compared to, for example, a scripting language.

The ideal syntax for expressing our acceptance tests could be a declarative, tabular structure such as a spreadsheet, or it could be something closer to a sequence of higher-level actions written in plain English. If we want to have our customer collaborate with developers on our acceptance tests, a full-blown programming language such as Java, C/C++, or C# is likely not an option. “Best tool for the job” means more than technically best, because the programmer’s job description also includes collaborating with the customer.

The acceptance TDD cycle

In its simplest form, the process of acceptance test-driven development can be expressed as the simple cycle illustrated by figure 1

Figure 1. The acceptance TDD cycle

This cycle continues throughout the iteration as long as we have more stories to implement, starting over again from picking a user story; then writing tests for the chosen story, then turning those tests into automated, executable tests; and finally implementing the functionality to make our acceptance tests pass.

In practice, of course, things aren’t always that simple. We might not yet have user stories, the stories might be ambiguous or even contradictory, the stories might not have been prioritized, the stories might have dependencies that affect their scheduling, and so on.

Step 1: Pick a user story

The first step is to decide which story to work on next. Not always an easy job; but, fortunately, most of the time we’ll already have some relative priorities in place for all the stories in our iteration’s work backlog. Assuming that we have such priorities, the simplest way to go is to always pick the story that’s on top of the stack—that is, the story that’s considered the most important of those remaining. Again, sometimes, it’s not that simple.

Generally speaking, the stories are coming from the various planning meetings held throughout the project where the customer informally describes new features, providing examples to illustrate how the system should work in each situation. In those meetings, the developers and testers typically ask questions about the features, making them a medium for intense learning and discussion. Some of that information gets documented on a story card (whether virtual or physical), and some of it remains as tacit knowledge. In those same planning meetings, the customer prioritizes the stack of user stories by their business value (including business risk) and technical risk (as estimated by the team).

There are times when the highest-priority story requires skills that we don’t possess, or we consider not having enough of. In those situations, we might want to skip to the next task to see whether it makes more sense for us to work on it. Teams that have adopted pair programming don’t suffer from this problem as often. When working in pairs, even the most cross-functional team can usually accommodate by adjusting their current pairs in a way that frees the necessary skills for picking the highest priority task from the pile.

The least qualified person

The traditional way of dividing work on a team is for everyone to do what they do best. It’s intuitive. It’s safe. But it might not be the best way of completing the task. Arlo Belshee presented an experience report at the Agile 2005 conference, where he described how his company had started consciously tweaking the way they work and measuring what works and what doesn’t. Among their findings about stuff that worked was a practice of giving tasks to the least qualified person.

There can be more issues to deal with regarding picking user stories, but most of the time the solution comes easily through judicious application of common sense. For now, let’s move on to the second step in our process: writing tests for the story we’ve just picked.

Step 2: Write tests for a story

With a story card in hand (or onscreen if you’ve opted for managing your stories online), our next step is to write tests for the story.

The first thing to do is, of course, get together with the customer. In practice, this means having a team member sit down with the customer (they’re the one who should own the tests, remember?) and start sketching out a list of tests for the story in question.

As usual, there are personal preferences for how to go about doing this, but current preference  is to quickly scram out a list of rough scenarios or aspects of the story we want to test in order to say that the feature has been implemented correctly. There’s time to elaborate on those rough scenarios later on when we’re implementing the story or implementing the acceptance tests. At this time, however, we’re only talking about coming up with a bulleted list of things we need to test—things that have to work in order for us to claim the story is done.

On timing

Especially in projects that have been going on for a while already, the customer and the development team probably have some kind of an idea of what’s going to get scheduled into the next iteration in the upcoming planning meeting. In such projects, the customer and the team have probably spent some time during the previous iteration sketching acceptance tests for the features most likely to get picked in the next iteration’s planning session. This means that we might be writing acceptance tests for stories that we’re not going to implement until maybe a couple of weeks from now. We also might think of missing tests during implementation, for example, so this test-writing might happen pretty much at any point in time between writing the user story and the moment when the customer accepts the story as completed.

Once we have such a rough list, we start elaborating the tests, adding more detail and discussing about how this and that should work, whether there are any specifics about the user interface the customer would like to dictate, and so forth. Depending on the type of feature, the tests might be a set of interaction sequences or flows, or they might be a set of inputs and expected outputs. Often, especially with flow-style tests, the tests specify some kind of a starting state, a context the test assumes is part of the system.

Other than the level of detail and the sequence in which we work to add that detail, there’s a question of when—or whether—to start writing the tests into an executable format. Witness step 3 in our process: automating the tests.

Step 3: Automate the tests

The next step once we’ve got acceptance tests written down on the back of a story card, on a whiteboard, in some electronic format, or on pink napkins, is to turn those tests into something we can execute automatically and get back a simple pass-or-fail result. Whereas we’ve called the previous step writing tests, we might call this step implementing or automating those tests.

In an attempt to avoid potential confusion about how the executable acceptance tests differ from the acceptance tests

We might turn Acceptance tests into an executable format by using a variety of approaches and tools. The most popular category of tools  these days seems to be what we calltable-based tools. Their premise is that the tabular format of tables, rows, and columns makes it easy for us to specify our tests in a way that’s both human and machine readable. Figure 2 presents an example of how we might draft an executable test for the first test  “Valid account number”.

Figure 2. Example of an executable test, sketched on a piece of paper

In figure 2, we’ve outlined the steps we’re going to execute as part of our executable test in order to verify that the case of an incoming support call with a valid account number is handled as expected, displaying the customer’s information onscreen. Our test is already expressed in a format that’s easy to turn into a tabular table format using our tool of choice—for example, something that eats HTML tables and translates their content into a sequence of method invocations to Java code according to some documented rules.

The inevitable fact is that most of the time, there is not such a tool available that would understand our domain language tests in our table format and be able to wire those tests into calls to the system under test. In practice, we’ll have to do that wiring ourselves anyway—most likely the developers or testers will do so using a programming language.

To summarize this duality of turning acceptance tests into executable tests, we’re dealing with expressing the tests in a format that’s both human and machine readable and with writing the plumbing code to connect those tests to the system under test.

On style

The example in figure 2 is a flow-style test, based on a sequence of actions and parameters for those actions. This is not the only style at our disposal, however. A declarative approach to expressing the desired functionality or business rule can often yield more compact and more expressive tests than what’s possible with flow-style tests.

Yet our goal should—once again—be to keep our tests as simple and to the point as possible, ideally speaking in terms of what we’re doing instead of how we’re doing it.

With regard to writing things down , there are variations on how different teams do this. Some start writing the tests right away into electronic format using a word processor; some even go so far as to write them directly in an executable syntax. Some teams run their tests as early as during the initial authoring session. Some people,  prefer to work on the tests alongside the customer using a physical medium, leaving the running of the executable tests for a later time. For example, agilists like to sketch the executable tests on a whiteboard or a piece of paper first, and pick up the computerized tools only when they got something  relatively sure won’t need to be changed right away.

The benefit is that we’re less likely to fall prey to the technology—Agilsts noticed that tools often steal too much focus from the topic, which we don’t want. Using software also has this strange effect of the artifacts being worked on somehow seeming more formal, more final, and thus needing more polishing up. All that costs time and money, keeping us from the important work.

In projects where the customer’s availability is the bottleneck, especially in the beginning of an iteration (and this is the case more often than not), it makes a lot of sense to have a team member do the possibly laborious or uninteresting translation step on their own rather than keep the customer from working on elaborating tests for other stories. The downside to having the team member formulate the executable syntax alone is that the customer might feel less ownership in the acceptance tests in general—after all, it’s not the exact same piece they were working on. Furthermore, depending on the chosen test-automation tool and its syntax, the customer might even have difficulty reading the acceptance tests once they’ve been shoved into the executable format dictated by the tool.

let’s consider a case where our test-automation tool is a framework for which we express our tests in a simple but powerful scripting language such as Ruby. Figure 3 highlights the issue with the customer likely not being as capable of feeling ownership of the implemented acceptance test compared to the sketch, which they have participated in writing. Although the executable snippet of Ruby code certainly reads nicely to a programmer, it’s not so trivial for a non-technical person to relate to.

Figure 3. Contrast between a sketch an actual, implemented executable acceptance test

Another aspect to take into consideration is whether we should make all tests executable to start with or whether we should automate one test at a time as we progress with the implementation. Some teams—and this is largely dependent on the level of certainty regarding the requirements—do fine by automating all known tests for a given story up front before moving on to implementing the story.

Some teams prefer moving in baby steps like they do in regular test-driven development, implementing one test, implementing the respective slice of the story, implementing another test, and so forth. The downside to automating all tests up front is, of course, that we’re risking more unfinished work—inventory, if you will—than we would be if we’d implemented one slice at a time. Agilists  preference is strongly on the side of implementing acceptance tests one at a time rather than try getting them all done in one big burst. It should be mentioned, though, that elaborating acceptance tests toward their executable form during planning sessions could help a team understand the complexity of the story better and, thus, aid in making better estimates.

Many of the decisions regarding physical versus electronic medium, translating to executable syntax together or not, and so forth also depend to a large degree on the people. Some customers have no trouble working on the tests directly in the executable format (especially if the tool supports developing a domain-specific language). Some customers don’t have trouble identifying with tests that have been translated from their writing. As in so many aspects of software development, it depends.

Regardless of our choice of how many tests to automate at a time, after finishing this step of the cycle we have at least one acceptance test turned into an executable format; and before we proceed to implementing the functionality in question, we will have also written the necessary plumbing code for letting the test-automation tool know what those funny words mean in terms of technology. That is, we will have identified what the system should do when we say “select a transaction” or “place a call”—in terms of the programming API or other interface exposed by the system under test.

To put it another way, once we’ve gotten this far, we have an acceptance test that we can execute and that tells us that the specified functionality is missing. The next step is naturally to make that test pass—that is, implement the functionality to satisfy the failing test.

Step 4: Implement the functionality

Next on our plate is to come up with the functionality that makes our newly minted acceptance test(s) pass. Acceptance test-driven development doesn’t say how we should implement the functionality; but, needless to say, it is generally considered best practice among practitioners of acceptance TDD to do the implementation using test-driven development.

In general, a given story represents a piece of customer-valued functionality that is split—by the developers—into a set of tasks required for creating that functionality. It is these tasks that the developer then proceeds to tackle using whatever tools necessary, including TDD. When a given task is completed, the developer moves on to the next task, and so forth, until the story is completed—which is indicated by the acceptance tests executing successfully.

In practice, this process means plenty of small iterations within iterations. Figure 4 visualizes this transition to and from test-driven development inside the acceptance TDD process.

Figure 4. The relationship between test-driven development and acceptance test-driven development

As we can see, the fourth step of the acceptance test-driven development cycle, implementing the necessary functionality to fix a failing acceptance test, can be expanded into a sequence of smaller TDD cycles of test-code-refactor, building up the missing functionality in a piecemeal fashion until the acceptance test passes.

While the developer is working on a story, frequently consulting with the customer on how this and that ought to work, there will undoubtedly be occasions when the developer comes up with a scenario—a test—that the system should probably handle in addition to the customer/developer writing those things down. Being rational creatures, we add those acceptance tests to our list, perhaps after asking the customer what they think of the test. After all, they might not assign as much value to the given aspect or functionality of the story as we the developers might.

At some point, we’ve iterated through all the tasks and all the tests we’ve identified for the story, and the acceptance tests are happily passing. At this point, depending on whether we opted for automating all tests up front  or automating them just in time, we either go back to Step 3 to automate another test or to Step 1 to pick a brand-new story to work on.

. Getting acceptance tests passing is intensive work.

 Acceptance TDD inside an iteration

A healthy iteration consists mostly of hard work. Spend too much time in meetings or planning ahead, and you’re soon behind the iteration schedule and need to de-scope . Given a clear goal for the iteration, good user stories, and access to someone to answer our questions, most of the iteration should be spent in small cycles of a few hours to a couple of days writing acceptance tests, collaborating with the customer where necessary, making the tests executable, and implementing the missing functionality with test-driven development.

As such, the four-step acceptance test-driven development cycle of picking a story, writing tests for the story, implementing the tests, and implementing the story is only a fraction of the larger continuum of a whole iteration made of multiple—even up to dozens—of user stories, depending on the size of your team and the size of your stories. In order to gain understanding of how the small four-step cycle for a single user story fits into the iteration, we’re going to touch the zoom dial and see what an iteration might look like on a time line with the acceptance TDD–related activities scattered over the duration of a single iteration.

Figure 5 is an attempt to describe what such a time line might look like for a single iteration with nine user stories to implement. Each of the bars represents a single user story moving through the steps of writing acceptance tests, implementing acceptance tests, and implementing the story itself. In practice, there could (and probably would) be more iterations within each story, because we generally don’t write and implement all acceptance tests in one go but rather proceed through tests one by one.

Figure 5. Putting acceptance test-driven development on time line

Notice how the stories get completed almost from the beginning of the iteration? That’s the secret ingredient that acceptance TDD packs to provide indication of real progress. Our two imaginary developers (or pairs of developers and/or testers, if we’re pair programming) start working on the next-highest priority story as soon as they’re done with their current story. The developers don’t begin working on a new story before the current story is done. Thus, there are always two user stories getting worked on, and functionality gets completed throughout the iteration.

So, if the iteration doesn’t include writing the user stories, where are they coming from? As you may know if you’re familiar with agile methods, there is usually some kind of a planning meeting in the beginning of the iteration where the customer decides which stories get implemented in that iteration and which stories are left in the stack for the future. Because we’re scheduling the stories in that meeting, clearly we’ll have to have those stories written before the meeting, no?

That’s where continuous planning comes into the picture.

Continuous planning

Although an iteration should ideally be an autonomous, closed system that includes everything necessary to meet the iteration’s goal, it is often necessary—and useful—to prepare for the next iteration during the previous one by allocating some amount of time for pre-iteration planning activities.  Suggestions regarding the time we should allocate for this continuous planning range from 10–15% of the team’s total time available during the iteration. As usual, it’s good to start with something that has worked for others and, once we’ve got some experience doing things that way, begin zeroing in on a number that seems to work best in our particular context.

In practice, these pre-iteration planning activities might involve going through the backlog of user stories, identifying stories that are most likely to get scheduled for the next iteration, identifying stories that have been rendered obsolete, and so forth. This ongoing pre-iteration planning is also the context in which we carry out the writing of user stories and, to some extent, the writing of the first acceptance tests. The rationale here is to be prepared for the next iteration’s beginning when the backlog of stories is put on the table. At that point, the better we know our backlog, the more smoothly the planning session goes, and the faster we get back to work, crunching out valuable functionality for our customer.

By writing, estimating, splitting if necessary, and prioritizing user stories before the planning meeting, we ensure quick and productive planning meetings and are able to get back to delivering valuable features sooner.

It would be nice if we had all acceptance tests implemented (and failing) before we start implementing the production code. That is often not a realistic scenario, however, because tests require effort as well—they don’t just appear from thin air—and investing our time in implementing the complete set of acceptance tests up front doesn’t make any more sense than big up-front design does in the larger scale. It is much more efficient to implement acceptance tests as we go, user story by user story.

Teams that have dedicated testing personnel can have the testing engineers work together with the customer to make acceptance tests executable while developers start implementing the functionality for the stories.  A hazard is  that most teams, however, are much more homogeneous in this regard and participate in writing and implementing acceptance tests together, with nobody designated as “the acceptance test guy.”

The process is largely dependent on the availability of the customer and the test and software engineers. If your customer is only onsite for a few days in the beginning of each iteration, you probably need to do some trade-offs in order to make the most out of those few days and defer work that can be deferred until after the customer is no longer available. Similarly, somebody has to write code, and it’s likely not the customer who’ll do that; software and test engineers need to be involved at some point.

We start from those stories we’ll be working on first, of course, and implement the user story in parallel with automating the acceptance tests that we’ll use to verify our work. And, if at all possible, we avoid having the same person implement the tests and the production code in order to minimize our risk of human nature playing its tricks on us.

Again, we want to keep an eye on putting too much up-front effort in automating our acceptance tests—we might end up with a huge bunch of tests but no working software. It’s much better to proceed in small steps, delivering one story at a time. No matter how valuable our acceptance tests are to us, their value to the customer is negligible without the associated functionality.

The mid-iteration sanity check

Agilists like to have an informal sanity check in the middle of an iteration. At that point, we should have approximately half of the stories scheduled for the iteration running and passing. This might not be the case for the first iteration, due to having to build up more infrastructure than in later iterations; but, especially as we get better at estimating our stories, it should always be in the remote vicinity of having 50% of the stories passing their tests.

Of course, we’ll be tracking story completion throughout the iteration. Sometimes we realize early on that our estimated burn rate was clearly off, and we must adjust the backlog immediately and accordingly. By the middle of an iteration, however, we should generally be pretty close to having half the stories for the iteration completed. If not, the chances are that there’s more work to do than the team’s capacity can sustain, or the stories are too big compared to the iteration length.

A story’s burn-down rate is constantly more accurate a source of prediction than an inherently optimistic software developer. If it looks like we’re not going to live up to our planned iteration content, we decrease our load.

Decreasing the load

When it looks like we’re running out of time, we decrease the load. We don’t work harder (or smarter). We’re way past that illusion. We don’t want to sacrifice quality, because producing good quality guarantees the sustainability of our productivity, whereas bad quality only creates more rework and grinds our progress to a halt. We also don’t want to have our developers burn out from working overtime, especially when we know that working overtime doesn’t make any difference in the long run. Instead, we adjust the one thing we can: the iteration’s scope—to reality. In general, there are three ways to do that: swap, drop, and split. Tom DeMarco and Timothy Lister have done a great favor to our industry with their best-selling books Slack (DeMarco; Broadway, 2001) and Peopleware (DeMarco, Lister; Dorset House, 1999), which explain how overtime reduces productivity.

Swapping stories is simple. We trade one story for another, smaller one, thereby decreasing our workload. Again, we must consult the customer in order to assure that we still have the best possible content for the current iteration, given our best knowledge of how much work we can complete.

Dropping user stories is almost as straightforward as swapping them. “This low-priority story right here, we won’t do in this iteration. We’ll put it back into the product backlog.” But dropping the lowest-priority story might not always be the best option, considering the overall value delivered by the iteration—that particular story might be of low priority in itself, but it might also be part of a bigger whole that our customer cares about. We don’t want to optimize locally. Instead, we want to make sure that what we deliver in the end of the iteration is a cohesive whole that makes sense and can stand on its own.

The third way to decrease our load, splitting, is a bit trickier compared to dropping and swapping

Splitting stories

How do we split a story we already tried hard to keep as small as possible during the initial planning game? In general, we can split stories by function or by detail (or both). Consider a story such as “As a regular user of the online banking application, I want to optionally select the recipient information for a bank transfer from a list of most frequently and recently used accounts based on my history so that I don’t have to type in the details for the recipients every time.”

Splitting this story by function could mean dividing the story into “…from a list of recently used accounts” and “…from a list of most frequently used accounts.” Plus, depending on what the customer means by “most frequently and recently used,” we might end up adding another story along the lines of “…from a weighted list of most frequently and recently used accounts” where the weighted list uses an algorithm specified by the customer. Having these multiple smaller stories, we could then start by implementing a subset of the original, large story’s functionality and then add to it by implementing the other slices, building on what we have implemented for the earlier stories.

Splitting it by detail could result in separate stories for remembering only the account numbers, then also the recipient names, then the VAT numbers, and so forth. The usefulness of this approach is greatly dependent on the distribution of the overall effort between the details—if most of the work is in building the common infrastructure rather than in adding support for one more detail, then splitting by function might be a better option. On the other hand, if a significant part of the effort is in, for example, manually adding stuff to various places in the code base to support one new persistent field, splitting by detail might make sense.

Regardless of the chosen strategy, the most important thing to keep in mind is that, after the splitting, the resulting user stories should still represent something that makes sense—something valuable—to the customer.

PEARL VIII : Agile Test Automation : Application of agile development principles to Test Automation

PEARL VIII :  Agile Test Automation : Application of agile development principles to Test Automation

Introduction

Today, more and more software development companies are shifting to an agile development approach from traditional waterfall development to keep pace with the current trends.To meet the demands of agile environment, several companies have adopted test automation which resolves some of the issues faced in manual testing and delivers faster results.

Even though test automation provides assistance in many ways, there are several challenges in test automation which can turn out to be a nightmare if it is adopted without appropriate brainstorming and analysis. Here’s a complete list of approaches to make test automation work successfully in agile environment.

The Development methodology agile team follows when creating automated tests is very similar to the process they follow for creating the software being tested. It involves a fair bit of design, coding and testing of its own to get it working correctly. So just like the application itself, automated tests are best developed incrementally – adding new tests and features to the automation framework over several different sprints. It’s important not to aim for the perfect “it-can-do-everything” test framework right at the start, as it would never materialize. Balance the cost vs. ROI well and come up with a bare minimum working solution at the start.

As more companies move from a traditional waterfall software development approach to an agile Methodology, suppliers are offering more test automation tools and services. Automated tests can provide faster feedback than a manual test, reducing rework and long feedback cycles.

Manual testing, particularly manual exploratory testing, is still important. However,  Agile teams typically find that the fast feedback afforded by automated regression is  a key to detecting problems quickly, thus reducing risk and rework.

Research firm Forrester expects testing-as-a-service (TaaS) to emerge as a managed test service from an increase in test automation, as large suppliers such as HP seek to support customers with test automation tools and outsourcing options, including security testing and services for SAP. Better test automation is intrinsic to agile development

Working tests, just like working software, are useful; they build confidence and get everyone excited about the progress being made. Successes, even small ones, make it easier to bring everyone on-board – especially when the test automation solution team have created actually runs and proves to be of real value to the team.

Agile Test team can’t set up automated testing tools without team members who can write code.  Automating tests is software development. It must be done with the same care and thought that goes into writing production code.This poses a significant hurdle for testers who lack coding skills. That challenge is best met by mastering those skills, not by shying away from projects that demand them

Better software is the result of running the right tests and continually re-evaluating which tests are the right ones,

Automated unit tests check the behavior of individual functions/methods and object interactions. They’re to run often, and provide feedback in minutes. Automated acceptance tests usually check the behavior of the system end-to-end. (Although,  sometimes they bypass the GUI, checking the underlying business logic.) They’re  typically run on checked in code on an ongoing basis, providing feedback in an hour  or so. Agile projects favor automated tests because of the rapid feedback they provide.

Agile Testing Quadrant

TheAgileTestingQuadrants.png

In the agile testing quadrant depicted above, The order in which quadrants numbered  has no relationship to when the different types of testing are done. Most projects would start with Q2 tests, because those are where you get the examples that turn into specifications and tests that drive coding, along with prototypes and the like. The quadrants are a taxonomy to help teams plan their testing and make sure they have all the resources they need to accomplish it.The timing of the various types of tests depends on the risks of each project, the customers’ goals for the product, whether the team is working with legacy code or on a greenfield project, and when resources are available to do the testing.

The lower left quadrant represents test-driven development, which is a core agile development practice. The quadrants on the left include tests that support the team as it develops the product.The testing done in Quadrants 1 and 2 are more requirements specification and design aids than what we typically think of as testing.

Unit tests verify functionality of a small subset of the system, such as an object or method. Component tests verify the behavior of a larger part of the system, such as a group of classes that provide some service [Meszaros, 2007]. Both types of tests are usually automated with a member of the xUnit family of test automation tools. We refer to these tests as programmer tests, developerfacing tests, or technology-facing tests. They enable the programmers to measure what Kent Beck has called the internal quality of their code [Beck, 1999].
A major purpose of Quadrant 1 tests is test-driven development (TDD) or test-driven design. The process of writing tests first helps programmers design their code well. These tests let the programmers confidently write code to deliver a story’s features without worrying about making unintended changes to the system. They can verify that their design and architecture decisions are appropriate. Unit and component tests are automated and written in the same programming language as the application. A business expert probably couldn’t understand them by reading them directly, but these tests aren’t intended for customer use. In fact, internal quality isn’t negotiated with the customer; it’s defined by the programmers. Programmer tests are normally part of an automated process that runs with every code check-in, giving the team instant, continual feedback about their internal quality

The tests in Quadrant 2 also support the work of the development team, but at a higher level. These business-facing tests, also called customer-facing tests and customer tests, define external quality and the features that the customers want.

They describe the details of each story. Business-facing tests run at a functional level, each one verifying a business satisfaction condition. They’re written in a way business experts can easily understand using the business domain language. In fact, the business experts use these tests to define the external quality of the product and usually help to write them.

One of the most important purposes of tests in these two quadrants (Q1 and Q2) is to provide information quickly and enable fast troubleshooting. They must be run frequently in order to give the team early feedback in case any behavior changes unexpectedly. All of these tests should be run as part of an automated continuous integration, build, and test process.

Technology-facing tests in Quadrant 4 are intended to critique product characteristics such as performance, robustness, and security. The types of tests that fall into the fourth quadrant are just as critical to agile development as to any type of software development. These tests are technology-facing. Creating and running these tests might require the use of specialized tools and additional expertise.

Quadrant 4 Automation tools

Native database tools

  •  SQL, data import tools
  • Shell scripting 
  • Monitoring tools examples
    • jConsole
      • Application bottlenecks, memory leaks
    • jProfiler
      • Database and bean usage
  • Commercial load test tools
    • Loadrunner
    • Silk Performer
  • Open source test tools
    • jMeter
    • The Grinder
    • jUnitPerf
  • Performance test providers
  • Multiple sites

The short iterations of agile development give your team a chance to learn and experiment with the different testing quadrants

Test Automation Backlog

Maintain a test automation backlog for the project that contains all needed automation tasks and identified improvements. Then target a few items from the backlog every sprint, in no time team will start to see the new regression test suite taking shape. Occasionally, stories from the test automation backlog may require dedicated developer time to implement and consequently some buy-in from the product owner in order to proceed. However, it should not be difficult to convince the product owner about the value of such stories if everyone on team is committed to quality. A test automation backlog could contain a prioritized list of items such as:

  • Parameterize the test environment for test execution.
  • Integrate with Continuous Integration.
  • Enhance reporting mechanism.
  • Provide an option to attach error logs in notification emails.
  • Collect performance metrics for workflow scenarios.
  • Add tests to check for concurrent execution of critical test cases.

Agile test automation pyramid

TA-pyramid 2 Figure 1

In the traditional Test Automation Pyramid view, most if not all of the effort was in developing UI-centric functional tests that explored the application via the GUI. There might be some lower-level tests and a few unit tests, but teams mostly staid at the upper tier.

The agile test automation pyramid is a strategy that attempts to alter the Test automation pyramid and provide the Agile Test automation pyramid depicted in the figure above.

Agile testing relies more on automation. It requires a much greater contribution from developers. And it has a different basic philosophy – to prevent bugs.

The first change is taking a whole-team view. Instead of the testers being responsible for testing AND writing all of the test automation, it becomes a whole-team responsibility. The developers take most of the ownership for unit-level automation, but testers can operate here as well.

At the base of the test automation pyramid is unit testing. Unit testing should be the foundation of a solid test automation strategy and as such represents the largest part of the pyramid.(50 % to 60 % – Unit Tests)

The upper tier focuses on limited GUI-based automation tests. Usually, these are longer running, core customer usage workflows that are best implemented at this level.

Automated user interface testing (0% to 10%) is placed at the top of the test automation pyramid because we want to do as little of it as possible.

The testers typically operate on these, but there are very few tests within this tier. And remember that the developers can operate here as well. The two layers are met by middle-tier automation. This is often the domain of the open source ATDD/BDD tools, such as: FitNesse, Cucumber, JBehave, Robot Framework, and others. The middle tier is the Acceptance Test tier / API Layer tests are performed.

One key thing to note is that traditional automation was often a one-tool operation, with Mercury/HP commercial tooling (Quick Test Professional or QTP) leading that space.  The agile approach is tool agnostic, but also aggregates tools that are appropriate for each layer. Therefor no “one size fits all” thinking is allowed. For example, these are common used tools at each tier:

  1. UI tier:  Selenium, Watir, or traditional QTP
  2. Middle tier:  FitNesse, Robot Framework, and Cucumber
  3. Unit tier:  xUnit family variants for example JUnit or NUnit for Java and .Net respectively

The other consideration is that there are extensions to many of these. For example, both Robot Framework and Cucumber have Selenium plug-ins so that they can ‘drive’ the UI as well as the middle tier. This implies that the middle tier tooling, and automated tests for that matter, can extend or blend into the lower and upper tiers.

Rather than investing in extensive, heavyweight step-by-step manual test scripts in  Word or a test management tool, we capture expectations in a format supported by  automated test frameworks like FIT/Fitnesse. The test could be executed manually,  but more importantly that same test artifact becomes an automated test when the  programmers write a fixture to connect the test to the software under test.

Test-driven development (TDD)

One key benefit of agile software development is the ability to make changes to the code quickly and easily. As the team embraces changing requirements as a part of a living, breathing system, they need to know that their changes will not cause a domino effect of broken code. Ideally, the design of the system will be so simple and well isolated that any possible effects of a change will be readily apparent. In the real world, this is not always the case, especially when working with legacy systems. In this case, something must be implemented to ensure that if a change affects other areas of the system, there will be an immediate red-flag, rather than having to chase down the issues later.

After experimenting with different ways to enable this flexibility, agile teams hit upon the practice of Test Driven Development, which involves producing automated unit tests for production code before you write that production code. Instead of writing tests afterward (or, more typically, never writing those tests), you always begin with a unit test. For every small chunk of functionality in production code, you first build and run a small, focused test that specifies and validates what the code will do. This test might not even compile, at first, because all of the classes and methods it requires may not yet exist. Nevertheless, it functions as an executable specification. You then get it to compile with minimal production code, so that you can run it and watch it fail. You then produce exactly as much code as will enable that test to pass. Sometimes you expect it to fail, and it passes, which is also useful information.

This technique feels odd, at first, to quite a few programmers who try it. It’s a bit like rock climbers inching up a rock wall, placing anchors in the wall as they go. Why go to all this trouble? Surely it slows you down considerably. The answer is that it only makes sense if you end up relying heavily and repeatedly on those anchors (unit tests) later. Those who practice Test Driven Development regularly claim that the benefits of those unit tests more than pay back the effort required to write them.

For Test-First work, you will typically use one of the xUnit family of automated unit test frameworks (JUnit for Java, NUnit for C#, etc). These frameworks make it quite straightforward to create, run, organize, and manage large suites of unit tests. Test Driven Development has grown to be such a useful tool that it is well integrated into most of the major IDEs, including Visual Studio and Eclipse.

Test-driven development (TDD) is an advanced technique of using automated unit tests to drive the design of software and force decoupling of dependencies. The result of using this practice is a comprehensive suite of unit tests that can be run at any time to provide feedback that the software is still working. This technique is heavily emphasized by those using Agile development methodologies

The motto of test-driven development is “Red, Green, Refactor.”

  • Red: Create a test and make it fail.
  • Green: Make the test pass by any means necessary.
  • Refactor: Change the code to remove duplication in your project and to improve the design while ensuring that all tests still pass.

The Red/Green/Refactor cycle is repeated very quickly for each new unit of code.

Follow these steps (slight variations exist among TDD practitioners):

  1. Understand the requirements of the story, work item, or feature that you are working on.
  2. Red: Create a test and make it fail.
    1. Imagine how the new code should be called and write the test as if the code already existed. You will not get IntelliSense because the new method does not yet exist.
    2. Create the new production code stub. Write just enough code so that it compiles.
    3. Run the test. It should fail. This is a calibration measure to ensure that your test is calling the correct code and that the code is not working by accident. This is a meaningful failure, and you expect it to fail.
  3. Green: Make the test pass by any means necessary.
    1. Write the production code to make the test pass. Keep it simple.
    2. Some advocate the hard-coding of the expected return value first to verify that the test correctly detects success. This varies from practitioner to practitioner.
    3. If you’ve written the code so that the test passes as intended, you are finished. You do not have to write more code speculatively. The test is the objective definition of “done.” The phrase “You Ain’t Gonna Need It” (YAGNI) is often used to veto unnecessary work. If new functionality is still needed, then another test is needed. Make this one test pass and continue.
    4. When the test passes, you might want to run all tests up to this point to build confidence that everything else is still working.
  4. Refactor: Change the code to remove duplication in your project and to improve the design while ensuring that all tests still pass.
    1. Remove duplication caused by the addition of the new functionality.
    2. Make design changes to improve the overall solution.
    3. After each refactoring, rerun all the tests to ensure that they all still pass.
  5. Repeat the cycle. Each cycle should be very short, and a typical hour should contain many Red/Green/Refactor cycles.

Characteristics of a Good Unit Test

A good unit test has the following characteristics.

  • Runs fast, runs fast, runs fast. If the tests are slow, they will not be run often.
  • Separates or simulates environmental dependencies such as databases, file systems, networks, queues, and so on. Tests that exercise these will not run fast, and a failure does not give meaningful feedback about what the problem actually is.
  • Is very limited in scope. If the test fails, it’s obvious where to look for the problem. Use few Assert calls so that the offending code is obvious. It’s important to only test one thing in a single test.
  • Runs and passes in isolation. If the tests require special environmental setup or fail unexpectedly, then they are not good unit tests. Change them for simplicity and reliability. Tests should run and pass on any machine. The “works on my box” excuse doesn’t work.
  • Often uses stubs and mock objects. If the code being tested typically calls out to a database or file system, these dependencies must be simulated, or mocked. These dependencies will ordinarily be abstracted away by using interfaces.
  • Clearly reveals its intention. Another developer can look at the test and understand what is expected of the production code.

Rich feature sets provided by the automation framework would essentially reduce the effort put in by the development teams in the TDD workflow. Here are some of the aspects of automation frameworks that can pay rich dividends: Ease of tracking: Unit tests would be stored in a central repository (part of the automation framework) with all the development team members submitting their unit tests to it. Tests would be stored in hierarchical folder structure based on the product features and its components. With this, viewing and tracking of unit tests within and across teams would be smoother. Traceability of unit tests: Automation frameworks can ensure that each product requirement has a unit test associated with it. This ensures that all requirements are developed as part of the TDD process, thus avoiding development slippages.

Improving the development and review process: Automation infrastructure can facilitate tracking of all requirements by associating them with a developer and reviewer(s). This would ensure that development and review processes are organized. Unit test execution: A good automation framework ensures quick running of automated unit tests. The tests could be executed selectively for a component, set of features or the product itself. Reporting of test execution results: Results of the automated unit test for a component/ feature would be sent to the respective developer; this ensures quick reporting and cuts short the response time in refactoring unit tests from the developer. Automation infrastructure components: Automation frameworks could facilitate:

  • Cross-platform testing
  • Compatibility testing

xUnit frameworks

Developers may use computer-assisted testing frameworks, such as xUnit, to create and automatically run the test cases. Xunit frameworks provide assertion-style test validation capabilities and result reporting. These capabilities are critical for automation as they move the burden of execution validation from an independent post-processing activity to one that is included in the test execution. The execution framework provided by these test frameworks allows for the automatic execution of all system test cases or various subsets along with other features.

Fakes, mocks and integration tests

Unit tests are so named because they each test one unit of code. A complex module may have a thousand unit tests and a simple module may have only ten. The tests used for TDD should never cross process boundaries in a program, let alone network connections. Doing so introduces delays that make tests run slowly and discourage developers from running the whole suite. Introducing dependencies on external modules or data also turns unit tests into integration tests. If one module misbehaves in a chain of interrelated modules, it is not so immediately clear where to look for the cause of the failure.

When code under development relies on a database, a web service, or any other external process or service, enforcing a unit-testable separation is also an opportunity and a driving force to design more modular, more testable and more reusable code. Two steps are necessary:

  1. Whenever external access is needed in the final design, an interface should be defined that describes the access available.
  2. The interface should be implemented in two ways, one of which really accesses the external process, and the other of which is a fake or mock. Fake objects need do little more than add a message such as “Person object saved” to a trace log, against which a test assertion can be run to verify correct behaviour. Mock objects differ in that they themselves contain test assertions that can make the test fail, for example, if the person’s name and other data are not as expected.

Fake and mock object methods that return data, ostensibly from a data store or user, can help the test process by always returning the same, realistic data that tests can rely upon. They can also be set into predefined fault modes so that error-handling routines can be developed and reliably tested. In a fault mode, a method may return an invalid, incomplete or null response, or may throw an exception. Fake services other than data stores may also be useful in TDD: A fake encryption service may not, in fact, encrypt the data passed; a fake random number service may always return 1. Fake or mock implementations are examples of dependency injection.

A Test Double is a test-specific capability that substitutes for a system capability, typically a class or function, that the UUT depends on. There are two times at which test doubles can be introduced into a system: link and execution. Link time substitution is when the test double is compiled into the load module, which is executed to validate testing. This approach is typically used when running in an environment other than the target environment that requires doubles for the hardware level code for compilation. The alternative to linker substitution is run-time substitution in which the real functionality is replaced during the execution of a test cases. This substitution is typically done through the reassignment of known function pointers or object replacement.

Test doubles are of a number of different types and varying complexities:

  • Dummy – A dummy is the simplest form of a test double. It facilitates linker time substitution by providing a default return value where required.
  • Stub – A stub adds simplistic logic to a dummy, providing different outputs.
  • Spy – A spy captures and makes available parameter and state information, publishing accessors to test code for private information allowing for more advanced state validation.
  • Mock – A mock is specified by an individual test case to validate test-specific behavior, checking parameter values and call sequencing.
  • Simulator – A simulator is a comprehensive component providing a higher-fidelity approximation of the target capability (the thing being doubled). A simulator typically requires significant additional development effort.

A corollary of such dependency injection is that the actual database or other external-access code is never tested by the TDD process itself. To avoid errors that may arise from this, other tests are needed that instantiate the test-driven code with the “real” implementations of the interfaces discussed above. These are integration tests and are quite separate from the TDD unit tests. There are fewer of them, and they must be run less often than the unit tests. They can nonetheless be implemented using the same testing framework, such as xUnit.

Integration tests that alter any persistent store or database should always be designed carefully with consideration of the initial and final state of the files or database, even if any test fails. This is often achieved using some combination of the following techniques:

  • The TearDown method, which is integral to many test frameworks.
  • try…catch…finally exception handling structures where available.
  • Database transactions where a transaction atomically includes perhaps a write, a read and a matching delete operation.
  • Taking a “snapshot” of the database before running any tests and rolling back to the snapshot after each test run. This may be automated using a framework such as Ant or NAnt or a continuous integration system such as CruiseControl.
  • Initialising the database to a clean state before tests, rather than cleaning up after them. This may be relevant where cleaning up may make it difficult to diagnose test failures by deleting the final state of the database before detailed diagnosis can be performed.

Continuous Integration

CI is a software development practice, where Team members integrate their work frequently, usually with each check-in to the version control system . An automated server builds the system, runs all the unit tests and reports “green” or “red” to indicate if the build is stable/broken.  Building and Testing are done on a dedicated machine to eliminate “it works on my machine” dependencies.

Over the past three years, Ancestry.com, the world’s largest online family history resource, underwent a significant transformation in technology and infrastructure. Starting with the adoption of Agile development practices, the company evolved to a Continuous Delivery model that enables code release whenever the business requires it. Transitioning from large, weekly or bi-weekly software rollouts to smaller, incremental updates has allowed Ancestry.com to increase responsiveness and deliver new features to customers more quickly.

Continuous Integration features

  • Very High ROI. One of the first engineering practices an Agile teams should adopt.
  • Many open source and commercial tools available
  • Keep builds and tests fast (move slow running tests to a nightly build)
  • Adopt a “Stop the Line” mentality (all work stops until a broken build is fixed)
  • Test in a clone of production (Every environmental difference results in risk)

Continuous integration (CI) is the practice, in software engineering, of merging all developer working copies with a shared mainline several times a day. It was first named and proposed as part of extreme programming (XP). Its main aim is to prevent integration problems, referred to as “integration hell” in early descriptions of XP. CI can be seen as an intensification of practices of periodic integration advocated by earlier published methods of incremental and iterative software development, such as the Booch method. CI isn’t universally accepted as an improvement over frequent integration, so it is important to distinguish between the two as there is disagreement about the virtues of each.

CI was originally intended to be used in combination with automated unit tests written through the practices of test-driven development. Initially this was conceived of as running all unit tests and verifying they all passed before committing to the mainline. This helps avoid one developer’s work in progress breaking another developer’s copy. If necessary, partially complete features can be disabled before committing using feature toggles.

Later elaborations of the concept introduced build servers, which automatically run the unit tests periodically or even after every commit and report the results to the developers. The use of build servers (not necessarily running unit tests) had already been practised by some teams outside the XP community. Nowadays, many organisations have adopted CI without adopting all of XP.

In addition to automated unit tests, organisations using CI typically use a build server to implement continuous processes of applying quality control in general — small pieces of effort, applied frequently. In addition to running the unit and integration tests, such processes run additional static and dynamic tests, measure and profile performance, extract and format documentation from the source code and facilitate manual QA processes. This continuous application of quality control aims to improve the quality of software, and to reduce the time taken to deliver it, by replacing the traditional practice of applying quality control after completing all development. This is very similar to the original idea of integrating more frequently to make integration easier, only applied to QA processes.

In the same vein the practice of continuous delivery further extends CI by making sure the software checked in on the mainline is always in a state that can be deployed to users and makes the actual deployment process very rapid.

Twist at AutoTrader

Testing was growing unwieldy for AutoTrader.co.uk, the UK’s #1 automotive website, with multiple tools, lack of  automation, and increasing test maintenance. Enter Twist—the test automation platform that reduced testing effort  by 95%, achieved 89% test coverage, and minimized testing time by 75%. Furthermore, Twist removed the biggest  pain-point, by drastically reducing the overhead of test maintenance. With Twist, the team could add real value and  test new functionality at a rapid pace

Sustaining quality for a dynamic and complex application can end up in a vicious cycle of modifying tests, testing
new functionality, and optimizing existing tests. AutoTrader.co.uk is the UK’s most popular motoring website, and
the fourth busiest search engine, getting over 10 million unique visitors every month. With such high traffic
volumes, optimal test coverage is not just nice-to-have, but a critical necessity. However, testing such a complex,
and ever changing application was a maintenance-heavy, lengthy, and expensive exercise that was further
complicated by difficult testing tools. Auto Trader liked Twist’s support for writing test specifications in plain English,
automating robust regression tests, and integrating with continuous integration (CI) and source control products,
and decided to give it a try. Since 2008, Twist has dramatically reduced AutoTrader.co.uk’s testing effort and
increased test coverage. It has also improved collaboration between business analysts (BAs) and quality analysts
(QAs) on tests, by simplifying technically complex tests to readable English constructs. Twist’s highly refactored,
modular test suite now exhaustively tests the site in 12 minutes, and can easily keep pace with application changes.
Additionally, Twist’s usability makes it highly popular across the teams.

PEARL XVII : Multidimensional Testing Coverage Matrix

PEARL XVII:

In Agile Software Development Methodology, the Multi Dimensional Testing Coverage Matrix  of what needs to be tested should be defined early. The importance of this has grown exponentially the past few years, as the test coverage matrix has become increasingly complex.

Rapid prototyping and development techniques combined with Agile development methodologies are pushing the envelope on the best practice of testing early and testing often. Keeping pace with the quick development turn-around and shorter time to market and being adaptive to late changes in requirements requires effective management of quality process. The use of  multidimensional test coverage matrix which maps of test artifacts – test cases, test defects, test fixtures – mapped to the requirements – needs, features, use cases and supplementary requirements – as a QA scheduling and planning tool defined early in the project, though claimed to have been practiced, has been largely overlooked by the  software industry.

 View of Agile Delivery Life Cycle

View of Agile Delivery Life Cycle

User needs in an agile process are defined by a story (sometimes captured as use-cases and features) planned to be implemented iteratively. Work break down for development (in iterations) of these use-cases and features is defined in terms of tasks.

Test coverage can help in monitoring the quality of testing, and assist in directing the test generators to create test cases that cover areas that have not been tested before.

Get Better Structure of the Product: Understanding the end-to-end data flow of an  application helps to analyze the root cause of a defect. Split the complete  product into small segments that will help team to get a clear structure of a product as well  as flows and dependencies of each and every part of modules.

Depending on Windows/Mac/Linux/Unix platforms, different types of browser-based tools are important for agile testers to troubleshoot the  defects. For example, for Http(s) tracking tools – Parse Proxy, Http Watch, WebScarab,  Http Debugger Pro, Archillies, and log tracing tools- Winscp & Putty, XML SPY,etc. These dependencies shall be clearly depicted in the Multidimensional Test coverage Matrix.

The agile team needs to  develop good test scenarios for future automation, end-to-end testing and product  lifecycle analysis, to help in organizational process growth

Acceptance TDD (ATDD). Team can do TDD at the requirements level by writing  customer test, the equivalent of a function test or acceptance test in the traditional world. Acceptance TDD is often called behavior-driven development (BDD) or story test-driven development, where you first automate a failing story test, then driving the design via TDD until the story test passes (a story test is a customer acceptance test).

Requirement Types

There are several categories of requirements, these are: Stakeholder, HighLevel Business, Application, Data, Security, Test (Quality), Supplementary, and Change
Management. Some of these categories have a lower level of granularity associated with them,
Stakeholder Requirements
These usually come from the CEO of the company, or the body of the Board of Directors. These are very high-level. They usually focus on a major “problem” that they want solved. These can also include high-level expectations for: Performance (“instantaneous”), Availability (“24x7x365”) and Security (“totally secure”).

High-Level Business Requirements
High-level requirements will fulfill Stakeholder Requirements. They are traced directly from them.  High-Level business requirements trace to Application Requirements.
Application Requirements
Application requirements drive the development of the application. They fall into the following categories; Use Case Requirements, Business Requirements/Rules, Functional Requirements, Technical Requirements, User Interface Requirements, “Look and Feel” Requirements and Navigation Requirements. There are some very fine lines between these types, and the ultimate strategy may be to combine these categories as needed.
Use Case Requirements
These start in the form of Use-Case Documents, but can be broken down into individual
requirements. The use case describes the basic flow of the system, alternate paths through the system. Business Rules, Navigation and User Interface requirements are all derived from the use case, and are traced from it.
Business Requirements/Rules
A Business Requirement or Rule is something that is specific to the business. These are
sometimes listed as “non-functional” requirements, as they do not have any function to them. The rules exist whether or not a system exists. An example of this would be, for a transportation firm,“Trucks only can travel 55 mph,” and another would be “We accept no bid less than $100,000.” This is also likely a list of data that is required to be captured, with the data including the data items needed, such as Name, address, etc. If there is a business need for data to be in a certain format, or minimum or maximum length for a text field, for example, that too is captured here. Any edit, validation, or default value that serves a business purpose is also in this category. This list of business rules is often captured in a business catalog. These are all should be traced from the use case.
Functional Requirements
These are driven and traced from the Business Requirements and Rules. They describe how a business rule is carried out, or how data is captured. These are usually in the format of “system shall do this or do that.”
Technical Requirements
These are derived during the low level design process, and sometimes during development. They are driven from and traced from the Functional Requirements. They are not “user” features of the system, but describe the low-level required actions of the system, that only a designer or developer would know. For example, “When transaction is being processed, and an error of type XYZ occurs, a rollback of the database transaction is required.” They can also include details of an error message, and error handling requirements.
User Interface Requirements
These are driven from Functional and Use Case Requirements, are traced from them both, depending on where they were derived from. They include items such as screen layout, tab flow, mouse and keyboard use, what controls to use for what functions (e.g. radio button, pulldown list), and other “ease of use” issues.
“Look and Feel” Requirements
These are driven from Functional and Business Requirements, and are traced from them both, depending on where they were derived from. These can be categorized as the “non-functional”
User Interface type requirements. They deal with how fields look, regarding font, font size,
colors, graphics, etc.
Navigation Requirements
These are driven and traced from the Use Case, as the Use Case lists the flow of the system, and the Navigation Requirements depict how that flow will take place. They are usually presented in a storyboard format, and should show the screen flow of each use case, and every alternate flow. Additionally, they should state what happens to the data or transaction for each step, for example, what happens to the data entered on a previous screen if the user decides to “go back?” They include the various ways to get to all screens, and an application screen map should be one of the artifacts derived in this category of requirements.
Based on the above Application Requirements, the Data Requirements are derived, and are traced from the Application Requirements.
Data Requirements
Based on the data needed in the Business Rules, and taking other requirements types into consideration, the database platform, database tables, columns, data types, size and other database attributes can be created.
Security Requirements
These are traced from wherever they need to be traced from, as they encompass all requirements that have do directly with the application . They mainly will come from Business Rules.
Test (Quality) Requirements/Test Cases
The Test Cases are traced from all types of requirements, and can be percived as the all en compassing box that all requirements are inside of. This shows that all requirements shall have test cases traced to it. If that test case fails during testing, the team should be able to see what higher-level requirement would not be fulfilled.
Supplementary Requirements
Traced from Data and Application requirements, these refer to the technology infrastructure and operations type of requirements, such as hardware and network requirements. There are many Operations Requirements that must also be captured, in the categories of, Administration Requirements, Maintenance Requirements, Disaster/Recovery Requirements, Skill Set of personnel Requirements, Change Management Requirements, User Documentation Requirements, Training Requirements, and Demo/Presentation Requirements. Other nonfunctional requirements that do not fall into any other category would fall into this “catch-all” category.

Carefully Define Testing Coverage Matrix: With the help of multi-dimensional  matrix, define the coverage requirements for a vital part of the specifications process at  an early stage. The size of the coverage matrix has an important influence on staffing  needs and QA budgets. Defining the testing coverage matrix early benefits the management to distribute resources and have better control and analysis of what can be  done internally versus what should be outsourced.

Test coverage in the test plan states what requirements will be verified during what stages of the product life. Test Coverage is derived from design specifications and other requirements, such as safety standards or regulatory codes, where each requirement or specification of the design ideally will have one or more corresponding means of verification. Test coverage for different product life stages may overlap, but will not necessarily be exactly the same for all stages. For example, some requirements may be verified during Design Verification test, but not repeated during Acceptance test. Test coverage also feeds back into the design process, since the product may have to be designed to allow test access .

In order to reproduce production scenarios or customer-specific issues, development and test teams need to be able to change operating systems, browsers, and databases, or infrastructure parameters such CPU, memory, disk size, and network configuration. These scenarios must be addressed in the multidimensional test coverage matrix.

The use cases descriptions define the main success scenarios of the system. However, not every use case scenario ends in a success for the user. While elaborating the use-cases using the descriptive text to capture these alternate paths, new aspects of the systems come to light when exceptions are encountered (non-happy path behavior of the system is being captured).

Spence and Probasco refer to them as overloading the term requirements, a common point of confusion with Requirements Management. These may not be clear from the user needs and system features captured, but they are a very vital and essential aspect of the system behavior. To ensure that the system meets these requirements and for coverage to be effective, these have to be elicited clearly and traced completely. Alternate paths may also be captured using a usability (scenario) matrix . While the use cases are mapped against features (or cards) which are planned for the iteration, so can the use-cases, the use-case scenarios that stem from these and so on, cascading to the test cases (and test artifacts)

Note that the usage of the application flow, even though captured, could end-up varying the application flow based on the data . Supplementary requirements corresponding to the architectural requirements for the system cannot be mapped unless captured separately. These remain outside the functional requirements modeled by the use cases.

A sample list of business rules that have to be followed could be summarized.

When the mapping of the test case flows across functionality is carried, it becomes evident that the granularity of details falls short; when mapping the coverage of the test flows against the business rules  in the tabular representation.

For example, Based on the feature set as set out it is possible that any one of the flows used to ensure coverage of business requirement 1 could as well serve for business requirement 2. However, on closer scrutiny the test case flow that tests non-happy path scenario of business requirement rule 2 requires a further elaboration of the test flows against feature set.

Such gaps and inadequacies will come to light in a multidimensional test coverage matrix that is not granular and consequently, the test coverage falls short. Tracing every non functional, business and non-business requirement to test cases and scenarios should increase the confidence and coverage of testing and QA activities that can be performed. The usability flows and concrete test cases that cover the requirements and needs can be formulated and with each iteration, targeted test cases could be identified to be run or executed to address within the specific build.

Test Coverage matrix is really multi-dimensional and to be effective QA artifacts, they have to transcend the various phases of the development process – initiation, elaboration, construction and transition. Further, it has to be a “living” artifact, one that is updated with each iteration

Multi Dimensional Test coverage matrix example

Multi Dimensional Test coverage matrix example

Within iterations, a set of acceptance and regression tests have to be scheduled and performed to meet the exit criteria. Features and stories (in the form of cards) are planned in iterations in an agile methodology. With multi dimensional test coverage matrix and mapping of the test cases to features, use cases and defects, optimum test planning assuring the software quality within each build/release becomes effortless and convenient. By establishing effective multi dimensional test coverage matrix, helps to answer some of the following questions

1. What test cases should be selected to run for the current build – verify fixed defects, regression suite for the current fixes, apart from the base code smoke and build acceptance tests?

2. What impact does change in a specific set of non-functional and functional requirements have on the QA testing process in arriving at test estimates?

3. How can defects identified be mapped to the requirements that the iteration was scoped to achieve? And what surround testing and re-testing have to be carried out to validate before the defects can be closed out or new but related ones identified?

4. What change requests were brought about by the most recent build or iteration and what does this new change entail?

Testing coverage matrix also establishes tracking back to the exact requirements being implemented in the iteration improving coverage and confience in the quality process. This is of greater significance in agile projects where requirements documentation isn’t complete as requirements continue to evolve with each build or iteration. Agility ensures the process (and the product) is adaptive to changing requirements and using testing coverage for QA activities ensures that verification keeps up with these changes. impact on quality

Multi Dimensional Attributes of Test Coverage Matrix

Multi Dimensional Test coverage matrix defines a method for testing a complex software system, wherein the complex software system has interrelated system programs.

The method comprising the steps of:

  • Determining linkages between interrelated system programs using an  multidimensional test coverage matrix;
  • identifying a change in one or more of the interrelated system programs;
  • applying the multidimensional test coverage matrix such that it depicts information pertaining to a selected test scenario and test cases pertaining to the identified change,
  • And also to depict other test scenarios and test cases linked by two or more levels through at least one of dependencies and tributaries.
  • The test coverage matrix is adapted to depict linkages across three or more levels of dependency;

Multi dimensional Test coverage matrix enables to keep track of the complex relationships between various components of complex software system and to assist in the regression testing needed in the integration and subsequent modification of the complex software system.  it is used to store the relationship information between test cases (predetermined scenarios involving a series of specified transactions to test the behavior of one or more software components of the complex software system) and/or the individual components of the complex software system

Dependency Scenario

Dependency Scenario