Week 9 Advanced Testing

Question 1

How are inputs selected in random testing?

Answer 1

The input domain is identified and random numbers are mapped to that input domain according to a probabilistic distribution.

Question 2

What guarantees does random testing produce?

Answer 2

Probabilistic guarantees about the programs faultiness.

Question 3

What are the benefits of random testing?

Answer 3

Cheap, easy, calculates reliability using probability theory. Good alternative to ECP when lack of domain knowledge or data volume is large.

Question 4

What are the 4 types of input space sampling?

Answer 4

Uniform, Equispaced, Proportional sampling and adaptive.

Question 5

What are the 4 types of input space sampling?

Answer 5

Uniform, Equispaced, Proportional sampling and adaptive.

Question 6

What is uniform distribution?

Answer 6

Uniform random selection of the input space, every input has the same chance of being chosen.

Question 7

What is Equispaced sampling

Answer 7

Unsampled sections of the input space are equally spaced apart (every 10,100,1000 etc)

Question 8

What is proportional sampling

Answer 8

Sample according to the subdomain distribution

Question 9

Adaptive sampling

Answer 9

Utilise prior knowledge about what type of input is more likely to cause failures and focus on this.

Question 10

What is a test oracle?

Answer 10

A test oracle is a mechanism for determining whether a test has passed or failed.

Question 11

How is a test oracle used.

Answer 11

The real outputs of a test are compared to the output the oracle determines the test should really have.

Question 12

What is the oracle problem?

Answer 12

It is the question of how do you determine the correct output for a given input.

Question 13

What are solutions to the oracle problem.

Answer 13

1. Golden standard. (use results from an old equivalent system)
2. Parametric oracle.
3. Observer oracle. It knows normal output curve of the SUT, if curves don’t match its an exceptional output.
4. Assertion you know a fact to be a truth. Whenever the output violates this assumed truth its a wrong output.

Question 14

What are solutions to the oracle problem.

Answer 14

1. Golden standard. (use results from an old equivalent system)
2. Parametric oracle.
3. Observer oracle. It knows normal output curve of the SUT, if curves don’t match its an exceptional output.
4. Assertion you know a fact to be a truth. Whenever the output violates this assumed truth its a wrong output.

Question 15

What are some faults of random testing.

Answer 15

Corner faults escaping detection, uniform distribution can easily be wrong, non uniform has domain bias.

Question 16

What is the difference between partitioning testing vs proportional sampling.

Answer 16

There is no semantic meaning behind the division of the domain.

Question 17

What are the software metrics of random/partition testing?

Answer 17

P-measure, E-measure, F-measure, MTTF.

Question 18

What is P-measure

Answer 18

Probability of finding at least one failing test case on n number of tests.

Question 19

What is E-measure?

Answer 19

Expected number of triggered failures.

Question 20

What is F-measure?

Answer 20

Expected number of test cases required to trigger a failure.

Question 21

What is MTTF?

Answer 21

A maintenance metric that measures average amount of time a non-repairable asset operates before failure.

Question 22

Implications of random vs partition testing?

Answer 22

Perform equally if no. test cases in partition is proportional to subdomain.

E measure is same when failure rates of partitions are the same, better if buggy partition is selected.

P-measure? Better on partitions if the partitions are the same size and same no test cases.

Question 23

Which test input method was the best?

Answer 23

BVA was most effective.

Question 24

What is fuzz testing?

Answer 24

A negative software testing method which feeds malformed and unexpected input data to the system.

Question 25

What is negative software testing?

Answer 25

Where a negative result is desired in order to discover bugs or faults.

Question 26

What are the 5 fuzzing approaches?

Answer 26

Generic, Pattern based, Intelligient, Large volume and Exploit variant.

Question 27

What is generic fuzzing?

Answer 27

Crude random corruption of data.

Question 28

What is pattern based fuzzing?

Answer 28

Modify random data to conform to particular patterns.

Question 29

What is intelligient fuzzing?

Answer 29

Uses semi-valid data which may pass through initial lines of defense, this requires understanding the data format.

Question 30

What is large volume fuzzing?

Answer 30

Fuzz tests at large scale.

Question 31

What is exploit variant fuzzing?

Answer 31

Varies an exploitative input to take advantage of the same attack vector with a different input.

Question 32

What is mutation testing?

Answer 32

Mutation testing focuses on measuring the adequacy of test cases.
It is not a testing strategy.
Should be used in conjunction with traditional techniques.