Testing Flashcards
1
Q
Forms of Testing
A
- Manual testing: have a human test the code (print statements, unit tests, integration tests)
- Static analysis: analyze the program without executing it. Static analysis abstracts across all possible executions. The large amount of constraints often results in a state explosion, limiting scalability (compiler warnings, linters, clang checker)
- Dynamic analysis: analyze the program during a concrete execution. Dynamic analysis focuses on a single concrete run. The limited focus allows detailed analysis of a run but testing is incomplete. (white box/greybox/blackbox testing)
2
Q
8 “Laws” of Static Analysis
A
- Can’t check code you don’t see
- Can’t check code you can’t parse
- Not everything is implemented in C
- Not a bug
- Not all bugs matter
- False positives matter
- False negatives matter
- Annotations are extremely costly
3
Q
Limitations of Static Analysis
A
- Trade-off between soundness and completeness
- Soundness: find all bugs of type X
- Soundness is costly: checks are weak or complexity explodes
- Diminishing returns: initial analysis finds most bugs, spend exponentially more time on few remaining bugs
4
Q
Dynamic Analysis Forms
A
- Whitebox testing: symbolic execution
- Greybox testing: feedback-oriented fuzzing
- Blackbox testing: blackbox fuzzing
5
Q
Symbolic Execution
A
- Define set of conditions at code locations
-> SE determines triggering input - Testing: finding bugs in applications
-> Infer pre/post conditions and add assertions
-> Use symbolic execution to negate conditions
6
Q
A
7
Q
Laws of Dynamic Testing
A
- Can’t test code you don’t see
- Context is key
- Coverage matters
- Cycles are cheap but finite
8
Q
Fuzzing Classification: Input Generation
A
- Generational fuzzing:
-> Produces inputs according to input specification
-> Requires significant amount of manual work - Mutational fuzzing:
-> Generates inputs by mutating seeds
9
Q
Coverage-Guided Greybox Fuzzing
A
- Coverage instrumentation is lightweight and activates feedback
-> Program is instrumented to “collect” coverage
-> Coverage indicates executed functionality
-> Ties an input to explored code areas and program behavior - Counting activated edges gives good precision at low overhead but lacks context
- Mutations are no longer blind but tied to coverage (which mutations triggered new behavior?)
10
Q
AddressSanitizer: Policy
A
- Instrument each access, check for poison values
- Advantage: fast checks
- Disadvantage:
-> Large memory overhead
-> Still slow
-> Not a mitigation: Does not detect all bugs - Find:
-> buffer overflows
-> Limited support for use-after-free
-> Detects some spatial and some temporal memory safety violations
11
Q
MemorySanitizer
A
Detect uninitialized reads
12
Q
Undefined Behavior Sanitizer
A
- Detect undefined behavior
-> Unsigned/misaligned pointers
-> Signed integer overflow
-> Conversion between floating point types loading to overflow
-> Illegal use of NULL pointers
-> Illegal pointer arithmetic
13
Q
ThreadSanitizer
A
Finds data races between threads
14
Q
Fuzzing binaries
A
- Fuzzing binaries without coverage instrumentation falls back to blackbox fuzzing
- Alternate solution: Rewrite binaries to add coverage instrumentation
-> Rewrite dynamically (terrible performance)
-> Rewrite statically (more complex analysis, but much better performance)
15
Q
LibFuzzer
A
- In-process, coverage-guided, evolutionary fuzzing engine
- Follows “unit test” approach
