Apply these types of code coverage to squash most bugs

Code coverage

Code coverage, also called statement coverage, measures the percentage of the lines of code that have been executed when testing concludes. Imagine a simple piece of C# code, where bool is a simple value that can be true or false.

bool DoTwoThings(bool conditionA,  bool conditionB, bool conditionC)

    if (conditionA) {

    //do the first thing

    }

    if (conditionB || conditionC) {

    //do the second thing

    }

    return true;

}

With this code, the development team can achieve 100% statement coverage when it calls the function, as conditionA and conditionB are both true.

While it appears that 100% code coverage is easily achievable, code often includes exception handling for things that should never happen. Many developers aim for 80% statement coverage -- 80 out of 100 lines of code in unit tests. Yet, all too often, bugs show up when the app behaves differently from what the programmer expects -- perhaps executing, for example, the bottom if statement, but not the top one.

Branch and condition coverage

Branch coverage is another type of code coverage that measures all the possible branches or code behaviors executed by tests. In the example above, the app could execute four possible branches:

• both if statements;
• the top statement, but not the bottom;
• the bottom statement, but not the top; or
• neither if statement.

It's easier for a tester to assess code coverage than it is to assess branch coverage. Organizations that measure branch coverage with test tools often work on mission- and life-critical systems, where the cost of failure can include human life.

Also, branch coverage is complicated by the fact that conditions do not occur in isolation. For example, if conditionC is true and conditionB is false, the app calls the same lines of code, but experienced testers know that bugs can slip in this way. Condition coverage assesses all the different ways that code can get in.

Let's look at an example. This line of code divides the first number by the second:

int divideby(int numerator, int divisor) {

    return numerator/divisor;

}

We can achieve statement, branch and division coverage with a test when the function passes a numerator of 8 with a divisor of 2 and returns a correct result of 4. However, if we use 0 as a divisor, the software will throw an error. To cover these types of functions, testers must know the domain, the platform and where the errors come from.

Time and space input combinatorics coverage

The only way to get full test coverage for this one line of code is to test every possible combination of numbers -- from 0.0000001 to as many nines as your input field can handle -- for each input. The order in which you enter them might also matter, as a bug might only appear if you perform the operations in a certain sequence. It is also possible that a bug will only manifest when the application has to perform the same operation several times in a row.

Achieving full time and space input combinatorics coverage with a handheld calculator takes an unfathomably long time. For most software, the team must run a limited number of the most powerful tests in their arsenal rather than all of them, then use those results to draw conclusions about the quality of the system.

Most organizations are unable to achieve 100% test coverage with inputs like the example above because of the inherent practical and financial challenges of combinatorics testing. Instead, they come up with a representation of a system, like design or requirements documents, then try to achieve full coverage within that spectrum.

Requirements coverage

Organizations in regulated industries that seek full test coverage can turn to a requirements traceability matrix, which numbers all the application requirements and test cases on a spreadsheet. Developers can organize the requirements as columns and the test cases as rows, then use the matrix to ensure every requirement is tested at least once while minimizing the number of test cases.

These matrices help prove a system has been sufficiently tested, whether that means every feature is given just one check or dozens of validations of its various inputs and interactions in a complex dependency chain.

Though, if you test each feature only once, you haven't assessed a system; you might have simply performed a demonstration of it. As testing consultant Michael Bolton puts it, "If you touch each feature once, you haven't tested a system. You may have provided a demonstration of it."

Requirements often have hidden combinations. Take a car insurance application where the rating factor is calculated with 10 different variables, each with 10 possible choices. The sales rep must select one of 10 groups for age, one of 10 for deductible amount, one of 10 for driving history and so on.

While it appears that testing all 100 combinations will cover all the requirements, a bug might be linked to two or more variables. For example, a bug might only materialize when the driver is very young and has a bad driving record and a high deductible. So testing each of the 100 choices independently might not find this issue.

Requirements combinatorics coverage

When test vendors tout 100% requirements coverage, they typically mean one test per item. However, the National Institute of Science and Technology (NIST) observed that testers can find the majority of defects in the interactions of two variables.

In the car insurance example, all-pairs testing, also called pairwise testing, can help reveal defects. Testers use all-pairs tools to decide the various possible inputs and combinations that are worth testing. The tool groups them into different equivalence classes and then returns test ideas or test cases.

However, many bugs that reach production have nothing to do with requirements. A requirement can be as vague as, "when the screen is resized, everything should look right." But this type of result is untestable. So keep track of your test ideas, in addition to the requirements.

Test idea coverage

Publish an online spreadsheet or use your favorite task management tool to congregate test ideas that the whole team can view and edit. From there, a team member can reserve a test idea, execute it and mark it as complete.

This method changes regression testing from a predictable process to a list of emergent potential risks. Sort the list by importance, then run as many tests as possible until you're out of time. The test team gets a coverage mix of sorts, along with a cut line. With this process, instead of testing everything, the team can test the most important things and lobby for more time if they need it.