Testing Flashcards
1
Q
Forms of Testing
A
- Manual testing: have a human test the code (print statements, unit tests, integration tests)
- Static analysis: analyze the program without executing it. Static analysis abstracts across all possible executions. The large amount of constraints often results in a state explosion, limiting scalability (compiler warnings, linters, clang checker)
- Dynamic analysis: analyze the program during a concrete execution. Dynamic analysis focuses on a single concrete run. The limited focus allows detailed analysis of a run but testing is incomplete. (white box/greybox/blackbox testing)
2
Q
8 “Laws” of Static Analysis
A
- Can’t check code you don’t see
- Can’t check code you can’t parse
- Not everything is implemented in C
- Not a bug
- Not all bugs matter
- False positives matter
- False negatives matter
- Annotations are extremely costly
3
Q
Limitations of Static Analysis
A
- Trade-off between soundness and completeness
- Soundness: find all bugs of type X
- Soundness is costly: checks are weak or complexity explodes
- Diminishing returns: initial analysis finds most bugs, spend exponentially more time on few remaining bugs
4
Q
Dynamic Analysis Forms
A
- Whitebox testing: symbolic execution
- Greybox testing: feedback-oriented fuzzing
- Blackbox testing: blackbox fuzzing
5
Q
Symbolic Execution
A
- Define set of conditions at code locations
-> SE determines triggering input - Testing: finding bugs in applications
-> Infer pre/post conditions and add assertions
-> Use symbolic execution to negate conditions
6
Q
A
7
Q
Laws of Dynamic Testing
A
- Can’t test code you don’t see
- Context is key
- Coverage matters
- Cycles are cheap but finite
8
Q
Fuzzing Classification: Input Generation
A
- Generational fuzzing:
-> Produces inputs according to input specification
-> Requires significant amount of manual work - Mutational fuzzing:
-> Generates inputs by mutating seeds
9
Q
Coverage-Guided Greybox Fuzzing
A
- Coverage instrumentation is lightweight and activates feedback
-> Program is instrumented to “collect” coverage
-> Coverage indicates executed functionality
-> Ties an input to explored code areas and program behavior - Counting activated edges gives good precision at low overhead but lacks context
- Mutations are no longer blind but tied to coverage (which mutations triggered new behavior?)
10
Q
AddressSanitizer: Policy
A
- Instrument each access, check for poison values
- Advantage: fast checks
- Disadvantage:
-> Large memory overhead
-> Still slow
-> Not a mitigation: Does not detect all bugs - Find:
-> buffer overflows
-> Limited support for use-after-free
-> Detects some spatial and some temporal memory safety violations
11
Q
MemorySanitizer
A
Detect uninitialized reads
12
Q
Undefined Behavior Sanitizer
A
- Detect undefined behavior
-> Unsigned/misaligned pointers
-> Signed integer overflow
-> Conversion between floating point types loading to overflow
-> Illegal use of NULL pointers
-> Illegal pointer arithmetic
13
Q
ThreadSanitizer
A
Finds data races between threads
14
Q
Fuzzing binaries
A
- Fuzzing binaries without coverage instrumentation falls back to blackbox fuzzing
- Alternate solution: Rewrite binaries to add coverage instrumentation
-> Rewrite dynamically (terrible performance)
-> Rewrite statically (more complex analysis, but much better performance)
15
Q
LibFuzzer
A
- In-process, coverage-guided, evolutionary fuzzing engine
- Follows “unit test” approach
16
Q
FuzzGen
A
- Goal: Invoke API in right order with the right arguments
- Build complex, shared state to pass between calls
- Build dependency graph based on used functions
17
Q
What is a corpus in fuzzing?
A
a corpus refers to a collection of input data files used by the fuzzer to generate test cases. These input files serve as seeds for the fuzzer, which then mutates and manipulates them to explore different execution paths and trigger potential vulnerabilities in the target program.
