Test Adequacy

Question 1

Structural Analysis

Answer 1

Study how much of the structure of a program we have actually tested

Question 2

Dataflow Analysis

Answer 2

We study test adequacy from the point of view of the extent to which different data flows are managed

Question 3

Static Testing

Answer 3

Testing of a program through analysis of artefacts without actually executing the SUT, ex. requirements testing or code reviews.

Question 4

Dynamic Testing

Answer 4

Test a system via techniques that involve execution of the system under test. Black Box/White Box

Question 5

Black Box Testing

Answer 5

driven by specifications, assumes no knowledge of a system’s internal workings

Question 6

White Box Testing

Answer 6

Driven by knowledge of a system’s internal workings

Question 7

Gray Box Testing

Answer 7

combination of black box and white box testing techniques whereby a tester directs his testing based on partial knowledge about the internal workings of the SUT.

Question 8

Structural Testing

Answer 8

Determine test cases based on structure of the program. Asks “what is missing in our test suite”. Structural coverage increases confidence in thoroughness of testing.

Question 9

Structural Testing in Practice

Answer 9

• Create functional test suite first
• Measure structural coverage to identify what is missing
• Interpret unexecuted elements, which may be due to natural differences between specs and implementation or may reveal flaws
• Attractive cause automated. Coverage is a good measure of progress.

Question 10

Control Flow Graph (CFG)

Answer 10

A model of a single method that is closely related to source code. Directed graph where nodes represent regions of the source code and directed edges represent the possibility that program execution proceeds from the end of one region directly to the beginning of the other.

Question 11

Test Case

Answer 11

Set of inputs, execution conditions, and a pass/fail criterion

Question 12

Test Case Specification

Answer 12

A requirement to be satisfied by one or more actual test cases

Question 13

Test Suite

Answer 13

A set of test cases

Question 14

Test Obligation

Answer 14

Partial test case specification, requiring some property deemed important to thorough testing

Question 15

Adequacy Criterion

Answer 15

A predicate deciding whether or not a particular test suite adequately tests a particular program.

Question 16

Branch Testing

Answer 16

A test suite can achieve 100% statement coverage without executing all possible branches.

Question 17

Statement Coverage

Answer 17

Number of executed statements/Number of statements

Question 18

Statements vs Branches

Answer 18

Traversing all edges of a graph causes all nodes to be visited. Test suites that satisfy branch adequacy criterion for a program also satisfy the statement adequacy criterion for the same program.

Question 19

Condition Testing

Answer 19

All branches can still miss conditions.

Question 20

Branch Coverage exposes

Answer 20

faults in how a computation has been decomposed into cases:
- Intuitively attractive: Check programmer’s case analysis
- But only roughly: Groups cases with the same outcome

Question 21

Condition Coverage

Answer 21

Considers case analysis in more detail but also individual conditions in a compound Boolean expression.

Question 22

Basic Condition Testing

Answer 22

Adequacy Criterion = Each basic condition must be executed at least once. These can be satisfied without satisfying branch coverage.

Question 23

Branch And Condition Adequacy

Answer 23

Cover all conditions and decisions

Question 24

Compound Condition Adequacy

Answer 24

Cover all possible evaluations of compound conditions. Cover all branches of a decision tree

Question 25

Modified Condition/Decision (MC/DC) Motivation

Answer 25

Effectively test important combinations of conditions without exponential blowup in test suite size. Important combinations means each basic condition shown to independently affect the outcome of each decision.

Question 26

MCDC Requires

Answer 26

- For each basic condition C, two test cases: - Values of all evaluated conditions except C are the same - Compound condition as a whole evaluates to true for one and false for the other

Question 27

Path Adequacy

Answer 27

Consider combinations of decisions along paths. Adequacy criterion: each path must be executed at least once.

Question 28

If number of paths in a program with loops is unbounded

Answer 28

Simple criterion of path adequacy is usually impossible to satisfy

Question 29

Boundary Interior Path Testing

Answer 29

Group paths that only differ in the sub-path they follow when repeating the body of a loop. The set of paths from the root of the tree to each leaf is the required set of subpaths for boundary/interior coverage.

Question 30

Loop Boundary Adequacy

Answer 30

Variant of the boundary/interior criterion that treats loop boundaries similarly but is less stringent wrt other differences on paths.

Question 31

A test suite satisfies the loop boundary adequacy criterion if

Answer 31

for every loop: - In at least one test case the loop body is iterated 0 times - In at least one case the loop body is iterated once - In at least one case the loop body is iterates 2+ times

Question 32

Cyclomatic Number

Answer 32

Number of independent paths in the CFG

Question 33

Cyclomatic Adequacy

Answer 33

If e = #edges, n = #nodes, c = #connected components of a graph, it is: e - n + c for an arbitrary graph e - n + 2 for a CFG

Question 34

Cyclomatic Coverage

Answer 34

Counts the number of independent paths that have been exercised, relative to cyclomatic complexity

Question 35

The Infeasibility Problem

Answer 35

Sometimes criteria may not be satisfiable, this criterion requires execution of: - Statements that cannot be executed as a result of defensive programming/code reuse - Conditions that cannot be satisfied as a result of interdependent conditions - Paths that cannot be executed as a result of interdependent decision - Unreachable code is normal even in well maintained systems

Question 36

Def Use Pair

Answer 36

Associates a point in a program with a point where it is used. Def: Where a variable gets a value. Use: Extraction of a value from a variable

Question 37

Definition Clear Path

Answer 37

Path along the CFG from a definition to a use of the same variable without another definition of the variable between. This is what forms a def-use pair.

Question 38

Direct Data Dependence Graph

Answer 38

Made up of: Nodes - as in the CFG Edges: def-use (du) pairs labelled with variable name

Question 39

Control Dependence

Answer 39

Data Dependence: Where did these values come from? Control Dependence: Which statement controls if this statement executes?

Question 40

Dominators

Answer 40

Pre-Dominators can be used to make an intuitive notion of "controlling decision" precise. Post-Dominators are calculated in the reverse of the control flow graph, using a special exit node as the root.

Question 41

Data Flow Testing Motivation

Answer 41

- Middle ground in structural testing: statement and branch coverage don't test interactions. Distinguish important paths. - Intuition: Statements interact through data flow.

Question 42

DU Pair

Answer 42

A pair of definition and use for some variable, such that at least one DU path exists from the definition to the use.

Question 43

DU Path

Answer 43

A definition-clear path on the CFG starting from a definition to a use of a same variable. Definition clear: Value is not replaced on a path. Loops could create infinite DU paths between a def and a use.

Question 44

Data Flow Testing Adequacy Criteria

Answer 44

- All DU pairs: At least one test case each - All DU paths: Each simple (non looping) path is exercised by at least one test case - All definitions: For each def, there is at least one test case which exercises a DU pair containing it.