mutation testing
Mutation testing, also known as code mutation testing, is a form of white box testing in which testers change specific components of an application's source code to ensure a software test suite will be able to detect the changes. Changes introduced to the software are intended to cause errors in the program. Mutation testing is directed to ensure the quality of a software testing suite, not the applications the suite will go on to test.
Mutation testing is typically used to conduct unit tests. The goal is for the software test to be able to detect all mutated code. Changes (called mutations) can be implemented by making modifications to an existing line of code to a different value. For example, a statement could be deleted or duplicated, true or false expressions can be changed or other variables can be altered.
Mutations introduced to a program’s code will typically be small and contain one variable which causes a fault or bug. Multiple versions of the original program are then made, each with its own mutation, called mutants. The mutants are then tested, along with the original application. Once the tests are conducted, testers should then compare the results to the original program test.
If the tests with the mutants detect the same amount of issues as the test with the original program, then testers will know that either the code has failed to execute, or the software testing suite being used has failed to detect the mutations. The software testing suite should then be worked on to be more effective. The mutants can be kept and re-used in another code mutation test once the testing software has been worked on. If the test results from the mutants to the original programs are different, with the software test detecting the faults in the mutants, then the mutants can be discarded, or killed.
The software test suite can then be scored by using the mutation score. The mutation score is the percentage of killed mutants divided by the total number of mutants multiplied by 100.
Advantages & disadvantages
Advantages to code mutation include:
- The ability to ensure the identification of weak tests or code.
- Has a high level of error detection.
- The increased use of object-oriented frameworks and unit tests have brought about more mutation testing tools.
- Mutation scores can give organizations an idea of how useful their testing suite is.
Disadvantages to code mutation include:
-
The large number of mutants used to test a test suite can lead to a potentially confusing experience in testing each version, which may reduce mutation testing’s practicality without automation. - Because of the large number of mutants being tested, mutation testing can be time-consuming and pricey.
Mutation testing vs. regression testing
Regression testing can be easily confused with mutation testing at first glance. Regression testing is the process of testing new changes to a program to ensure that the older programming still works with new changes. Test department coders will develop code test scenarios that will test new units of code after being written.
Regression testing is used to test changes to a program to ensure a new version of an application which solves an old bug does not introduce another. Regression tests will look at applications to ensure that small changes to an application's code do not break the software. This is opposed to code mutation, which makes new changes to an application's code to test the test suite itself.
