Random Testing

Question 1

What is random testing? What do they provide?

Answer 1

Using program analysis to randomly sample new tests without user interaction
They provide a means of continuous testing

Question 2

What are 4 directions of random testing?

Answer 2

Fuzz testing
Feedback Directed Random Testing
Property Based Testing
Chaos Engineering

Question 3

Even though fuzz testing was historically naive, it was still ________

Answer 3

effective

Question 4

What are some questions you can ask to improve fuzzing?

Answer 4

Initial test suite required?
How are new tests generated
How does the result of the previous test affect test generation?
What kind of bugs can be found?

Question 5

What are 2 major ways to generate test inputs for fuzzing?

Answer 5

Generational - Create new inputs, need a input space model

Mutational - Modifies existing test suite

Question 6

What kind of models can you have in generation fuzz testing?

Answer 6

Grammar, binary file formats, wire protocols

Question 7

An example of generational fuzz testing is ______ ________. Specifications are provided through XML

Answer 7

Peach Fuzzer

Question 8

In mutational fuzz testing, you ______ new inputs from and existing set using heuristics

Answer 8

evolve

Question 9

What is coverage guided fuzzing?

Answer 9

Fuzzing where you evolve a test suite toward more coverage

Question 10

If we want to measure the following categories, what can we specify as a heuristic in our fuzzer?
Computational overhead
Memory Consumption
Energy Consumption
REST API invocations

Answer 10

Computational overhead - Count instruction frequency in coverage
Memory Consumption - Count allocated memory per allocation
Energy Consumption - Measure power consumption over tasks, etc
REST API invocations - Measure diversity of requests fed to server

Question 11

What are some common universal oracles?

Answer 11

Never crash
No undefined behaviour
No failures from dynamic analysis tools

Question 12

What is differential testing?

Answer 12

Feed input into N different implementations and vote

Question 13

What is metamorphic testing?

Answer 13

Identify key properties that enable correct results to be known relative to mutations

Question 14

What are a few challenges in fuzzing?

Answer 14

Highly structured inputs require more care
Making use of nuanced oracles can be challenging
It’s only effective at a whole program or single function level

Question 15

What is feedback directed random testing?

Answer 15

Generate a unit test following AAA

Use coverage feedback again to guide the process

Question 16

When would you use feedback directed random testing?

Answer 16

When testing something that is not based on input, such as an API or object oriented API

Question 17

What are some challenges in feedback directed random testing?

Answer 17

Defining good notions of coverage

Defining oracles