Mutation Testing

Question 1

What is fault seeding? What does it achieve?

Answer 1

Inserting representative or typical faults into your program. This allows you to measure how many bugs your test suite kills

Question 2

In mutation testing, how do you measure effectiveness?

Answer 2

killed/ # seeded

Question 3

What are 3 issues with fault seeding?

Answer 3

Finding representative faults
Putting in enough faults for the test to be meaningful
Forgetting to remove the faults afterwards

Question 4

What is a mutant?

Answer 4

A valid program that behaves differently than the original

Question 5

When does test t kill a mutant m?

Answer 5

If t produces a different outcome on m than the original program

Question 6

What are 4 classes of unhelpful mutants?

Answer 6

Still born (not compilable)
Trivial (killed by most test cases)
Equivalent (indistinguishable from original program)
Redundant (indistinguishable from other mutants)

Question 7

A mutant should contain only one _______ from the original program

Answer 7

change

Question 8

What kind of mutation mimics statement coverage? How about input classes?

Answer 8

Bomb()s

failOnZero(x)s

Question 9

Because of equivalent mutants, what is the update mutation score formula? What is this formula with duplicates taken into account?

Answer 9

killed/ (# mutants - # equivalent)

killed - #killed dupes) / (# mutants - # equivalent - # dupes

Question 10

Why are equivalent mutants equivalent?

Answer 10

• Mutation must be reachable
• Does the mutation create a different program state? (infection)
• Is the program state difference visible? (Propagation)

Equivalents mutants don’t have propagation, but they could infect the program or they might not be reachable

Question 11

Finding ________ mutants is one of the most expensive aspects of this type of testing

Answer 11

equivalent

Question 12

What are mutation operators? Why might they be language dependent?

Answer 12

Changes that are applied to produce mutants

Languages may or may not have co-routines, list comprehensions, hierarchies, etc

Question 13

REVIEW EDGE COVERAGE SECTION

Answer 13

REVIEW EDGE COVERAGE SECTION

Question 14

What is scalar variable replacement? Give an example

Answer 14

Replacing each variable use with another compatible variable. An example would be replacing a long with a short

Question 15

What are some interprocedural mutation operators?

Answer 15

Changing parameter values
Changing call target
Changing dependencies

Question 16

What are two reasons mutants are representative of all bugs?

Answer 16

Competent Programmer Hypothesis: Programmers tend to write code that is almost correct -> most of the time simple mutations reflect real bugs
Coupling Effect: tests that cover so many behaviors should detect both simple and complex errors

Question 17

What are higher order mutants? What is a major issue with them?

Answer 17

Mutants with multiple mutations operators. An issue is that we cannot isolate which change caused the bug

Question 18

What are 2 problems with mutation coverage?

Answer 18

Scale

Detecting Equivalence

Question 19

How can we deal with the scale problem in mutation coverage?

Answer 19

• Coverage filters (use statement coverage to prune mutants not in tests)
• Short circuiting tests
• Simultaneous testing (parallelism)
• Sampling

Question 20

What is the difference between a strongly killed mutant and a weakly killed mutant? What type of coverage do these conditions lead to?

Answer 20

Strongly Killed: The output of the test is different
Weakly killed: The internal state of the test is different due to the mutant. These tests don’t need propagation.

Strong Mutation Coverage and Weak Mutation Coverage

Question 21

Weak mutation coverage _______ many mutants, but is not very informative

Answer 21

prunes

Question 22

Why is mutation testing one of the strongest criteria?

Answer 22

Mimics different testing strategies (input partitioning, graph coverage)

Question 23

Although mutation testing has a massive number of criteria, why might it still not have the most tests?

Answer 23

The criteria is not exponential

Question 24

Shrinking the number of tests while preserving coverage _______ defect finding power

Answer 24

decreases

Question 25

Why is coverage important even if it cannot measure defect finding capability?

Answer 25

It tells you which portions of the program haven’t been tested

Question 26

Mutation testing correlates _____ with defect finding power, but it can be expensive

Answer 26

strongly