Week 9 Advanced Testing Flashcards
How are inputs selected in random testing?
The input domain is identified and random numbers are mapped to that input domain according to a probabilistic distribution.
What guarantees does random testing produce?
Probabilistic guarantees about the programs faultiness.
What are the benefits of random testing?
Cheap, easy, calculates reliability using probability theory. Good alternative to ECP when lack of domain knowledge or data volume is large.
What are the 4 types of input space sampling?
Uniform, Equispaced, Proportional sampling and adaptive.
What are the 4 types of input space sampling?
Uniform, Equispaced, Proportional sampling and adaptive.
What is uniform distribution?
Uniform random selection of the input space, every input has the same chance of being chosen.
What is Equispaced sampling
Unsampled sections of the input space are equally spaced apart (every 10,100,1000 etc)
What is proportional sampling
Sample according to the subdomain distribution
Adaptive sampling
Utilise prior knowledge about what type of input is more likely to cause failures and focus on this.
What is a test oracle?
A test oracle is a mechanism for determining whether a test has passed or failed.
How is a test oracle used.
The real outputs of a test are compared to the output the oracle determines the test should really have.
What is the oracle problem?
It is the question of how do you determine the correct output for a given input.
What are solutions to the oracle problem.
- Golden standard. (use results from an old equivalent system)
- Parametric oracle.
- Observer oracle. It knows normal output curve of the SUT, if curves don’t match its an exceptional output.
- Assertion you know a fact to be a truth. Whenever the output violates this assumed truth its a wrong output.
What are solutions to the oracle problem.
- Golden standard. (use results from an old equivalent system)
- Parametric oracle.
- Observer oracle. It knows normal output curve of the SUT, if curves don’t match its an exceptional output.
- Assertion you know a fact to be a truth. Whenever the output violates this assumed truth its a wrong output.
What are some faults of random testing.
Corner faults escaping detection, uniform distribution can easily be wrong, non uniform has domain bias.
What is the difference between partitioning testing vs proportional sampling.
There is no semantic meaning behind the division of the domain.
What are the software metrics of random/partition testing?
P-measure, E-measure, F-measure, MTTF.
What is P-measure
Probability of finding at least one failing test case on n number of tests.
What is E-measure?
Expected number of triggered failures.
What is F-measure?
Expected number of test cases required to trigger a failure.
What is MTTF?
A maintenance metric that measures average amount of time a non-repairable asset operates before failure.
Implications of random vs partition testing?
Perform equally if no. test cases in partition is proportional to subdomain.
E measure is same when failure rates of partitions are the same, better if buggy partition is selected.
P-measure? Better on partitions if the partitions are the same size and same no test cases.
Which test input method was the best?
BVA was most effective.
What is fuzz testing?
A negative software testing method which feeds malformed and unexpected input data to the system.
What is negative software testing?
Where a negative result is desired in order to discover bugs or faults.
What are the 5 fuzzing approaches?
Generic, Pattern based, Intelligient, Large volume and Exploit variant.
What is generic fuzzing?
Crude random corruption of data.
What is pattern based fuzzing?
Modify random data to conform to particular patterns.
What is intelligient fuzzing?
Uses semi-valid data which may pass through initial lines of defense, this requires understanding the data format.
What is large volume fuzzing?
Fuzz tests at large scale.
What is exploit variant fuzzing?
Varies an exploitative input to take advantage of the same attack vector with a different input.
What is mutation testing?
Mutation testing focuses on measuring the adequacy of test cases.
It is not a testing strategy.
Should be used in conjunction with traditional techniques.
