Mutation Testing Flashcards
What is fault seeding? What does it achieve?
Inserting representative or typical faults into your program. This allows you to measure how many bugs your test suite kills
In mutation testing, how do you measure effectiveness?
killed/ # seeded
What are 3 issues with fault seeding?
Finding representative faults
Putting in enough faults for the test to be meaningful
Forgetting to remove the faults afterwards
What is a mutant?
A valid program that behaves differently than the original
When does test t kill a mutant m?
If t produces a different outcome on m than the original program
What are 4 classes of unhelpful mutants?
Still born (not compilable)
Trivial (killed by most test cases)
Equivalent (indistinguishable from original program)
Redundant (indistinguishable from other mutants)
A mutant should contain only one _______ from the original program
change
What kind of mutation mimics statement coverage? How about input classes?
Bomb()s
failOnZero(x)s
Because of equivalent mutants, what is the update mutation score formula? What is this formula with duplicates taken into account?
killed/ (# mutants - # equivalent)
killed - #killed dupes) / (# mutants - # equivalent - # dupes
Why are equivalent mutants equivalent?
- Mutation must be reachable
- Does the mutation create a different program state? (infection)
- Is the program state difference visible? (Propagation)
Equivalents mutants don’t have propagation, but they could infect the program or they might not be reachable
Finding ________ mutants is one of the most expensive aspects of this type of testing
equivalent
What are mutation operators? Why might they be language dependent?
Changes that are applied to produce mutants
Languages may or may not have co-routines, list comprehensions, hierarchies, etc
REVIEW EDGE COVERAGE SECTION
REVIEW EDGE COVERAGE SECTION
What is scalar variable replacement? Give an example
Replacing each variable use with another compatible variable. An example would be replacing a long with a short
What are some interprocedural mutation operators?
Changing parameter values
Changing call target
Changing dependencies
What are two reasons mutants are representative of all bugs?
Competent Programmer Hypothesis: Programmers tend to write code that is almost correct -> most of the time simple mutations reflect real bugs
Coupling Effect: tests that cover so many behaviors should detect both simple and complex errors
What are higher order mutants? What is a major issue with them?
Mutants with multiple mutations operators. An issue is that we cannot isolate which change caused the bug
What are 2 problems with mutation coverage?
Scale
Detecting Equivalence
How can we deal with the scale problem in mutation coverage?
- Coverage filters (use statement coverage to prune mutants not in tests)
- Short circuiting tests
- Simultaneous testing (parallelism)
- Sampling
What is the difference between a strongly killed mutant and a weakly killed mutant? What type of coverage do these conditions lead to?
Strongly Killed: The output of the test is different
Weakly killed: The internal state of the test is different due to the mutant. These tests don’t need propagation.
Strong Mutation Coverage and Weak Mutation Coverage
Weak mutation coverage _______ many mutants, but is not very informative
prunes
Why is mutation testing one of the strongest criteria?
Mimics different testing strategies (input partitioning, graph coverage)
Although mutation testing has a massive number of criteria, why might it still not have the most tests?
The criteria is not exponential
Shrinking the number of tests while preserving coverage _______ defect finding power
decreases
Why is coverage important even if it cannot measure defect finding capability?
It tells you which portions of the program haven’t been tested
Mutation testing correlates _____ with defect finding power, but it can be expensive
strongly
