Test Adequacy Flashcards
Structural Analysis
Study how much of the structure of a program we have actually tested
Dataflow Analysis
We study test adequacy from the point of view of the extent to which different data flows are managed
Static Testing
Testing of a program through analysis of artefacts without actually executing the SUT, ex. requirements testing or code reviews.
Dynamic Testing
Test a system via techniques that involve execution of the system under test. Black Box/White Box
Black Box Testing
driven by specifications, assumes no knowledge of a system’s internal workings
White Box Testing
Driven by knowledge of a system’s internal workings
Gray Box Testing
combination of black box and white box testing techniques whereby a tester directs his testing based on partial knowledge about the internal workings of the SUT.
Structural Testing
Determine test cases based on structure of the program. Asks “what is missing in our test suite”. Structural coverage increases confidence in thoroughness of testing.
Structural Testing in Practice
- Create functional test suite first
- Measure structural coverage to identify what is missing
- Interpret unexecuted elements, which may be due to natural differences between specs and implementation or may reveal flaws
- Attractive cause automated. Coverage is a good measure of progress.
Control Flow Graph (CFG)
A model of a single method that is closely related to source code. Directed graph where nodes represent regions of the source code and directed edges represent the possibility that program execution proceeds from the end of one region directly to the beginning of the other.
Test Case
Set of inputs, execution conditions, and a pass/fail criterion
Test Case Specification
A requirement to be satisfied by one or more actual test cases
Test Suite
A set of test cases
Test Obligation
Partial test case specification, requiring some property deemed important to thorough testing
Adequacy Criterion
A predicate deciding whether or not a particular test suite adequately tests a particular program.
Branch Testing
A test suite can achieve 100% statement coverage without executing all possible branches.
Statement Coverage
Number of executed statements/Number of statements
Statements vs Branches
Traversing all edges of a graph causes all nodes to be visited. Test suites that satisfy branch adequacy criterion for a program also satisfy the statement adequacy criterion for the same program.
Condition Testing
All branches can still miss conditions.
Branch Coverage exposes
faults in how a computation has been decomposed into cases:
- Intuitively attractive: Check programmer’s case analysis
- But only roughly: Groups cases with the same outcome
Condition Coverage
Considers case analysis in more detail but also individual conditions in a compound Boolean expression.
Basic Condition Testing
Adequacy Criterion = Each basic condition must be executed at least once. These can be satisfied without satisfying branch coverage.
Branch And Condition Adequacy
Cover all conditions and decisions
Compound Condition Adequacy
Cover all possible evaluations of compound conditions. Cover all branches of a decision tree
Modified Condition/Decision (MC/DC) Motivation
Effectively test important combinations of conditions without exponential blowup in test suite size. Important combinations means each basic condition shown to independently affect the outcome of each decision.
MCDC Requires
- For each basic condition C, two test cases:
- Values of all evaluated conditions except C are the same
- Compound condition as a whole evaluates to true for one and false for the other
Path Adequacy
Consider combinations of decisions along paths. Adequacy criterion: each path must be executed at least once.
If number of paths in a program with loops is unbounded
Simple criterion of path adequacy is usually impossible to satisfy
Boundary Interior Path Testing
Group paths that only differ in the sub-path they follow when repeating the body of a loop. The set of paths from the root of the tree to each leaf is the required set of subpaths for boundary/interior coverage.
Loop Boundary Adequacy
Variant of the boundary/interior criterion that treats loop boundaries similarly but is less stringent wrt other differences on paths.
A test suite satisfies the loop boundary adequacy criterion if
for every loop:
- In at least one test case the loop body is iterated 0 times
- In at least one case the loop body is iterated once
- In at least one case the loop body is iterates 2+ times
Cyclomatic Number
Number of independent paths in the CFG
Cyclomatic Adequacy
If e = #edges, n = #nodes, c = #connected components of a graph, it is:
e - n + c for an arbitrary graph
e - n + 2 for a CFG
Cyclomatic Coverage
Counts the number of independent paths that have been exercised, relative to cyclomatic complexity
The Infeasibility Problem
Sometimes criteria may not be satisfiable, this criterion requires execution of:
- Statements that cannot be executed as a result of defensive programming/code reuse
- Conditions that cannot be satisfied as a result of interdependent conditions
- Paths that cannot be executed as a result of interdependent decision
- Unreachable code is normal even in well maintained systems
Def Use Pair
Associates a point in a program with a point where it is used. Def: Where a variable gets a value. Use: Extraction of a value from a variable
Definition Clear Path
Path along the CFG from a definition to a use of the same variable without another definition of the variable between. This is what forms a def-use pair.
Direct Data Dependence Graph
Made up of:
Nodes - as in the CFG
Edges: def-use (du) pairs labelled with variable name
Control Dependence
Data Dependence: Where did these values come from?
Control Dependence: Which statement controls if this statement executes?
Dominators
Pre-Dominators can be used to make an intuitive notion of “controlling decision” precise.
Post-Dominators are calculated in the reverse of the control flow graph, using a special exit node as the root.
Data Flow Testing Motivation
- Middle ground in structural testing: statement and branch coverage don’t test interactions. Distinguish important paths.
- Intuition: Statements interact through data flow.
DU Pair
A pair of definition and use for some variable, such that at least one DU path exists from the definition to the use.
DU Path
A definition-clear path on the CFG starting from a definition to a use of a same variable. Definition clear: Value is not replaced on a path. Loops could create infinite DU paths between a def and a use.
Data Flow Testing Adequacy Criteria
- All DU pairs: At least one test case each
- All DU paths: Each simple (non looping) path is exercised by at least one test case
- All definitions: For each def, there is at least one test case which exercises a DU pair containing it.
