Clear testing goals Flashcards
Software Metrics
System or standard of measurement which gives a value to some property
Coverage
A measure of proportion of a structure that a program, test case, test suite exercises.
List test coverage methods based on rigour from low to high.
- Statement coverage
- Decision coverage
- Condition Coverage
- Condition/Decision Coverage.
- Modified Condition/decision coverage.
What can analysis of structural coverage discover?
- Shortcomings in requirements-based test cases
- inadequacies in requirements
- dead code/deactivated code
- unintended functionalities.
Path coverage
Test suite achieves path coverage if it results in every path in the program taken at least once.
Path coverage v. Exhaustive testing
Not the same. Path coverage ensures each control flow path is seen once. not that all paths have seen all possible values.
Statement Coverage
Test suite achieves Statement coverage if it results in each statement in the program being run at least once.
Statement coverage for:
int abs(int x) {
int ans = x;
if x < 0 {
ans = -x
}
return ans;
}test case: abs(-4).
Hits every statement in the code but doesn’t hit every branch in the program.
Branch Coverage
Test suite achieves branch coverage if it results in each branch in the program being run at least once.
Branch coverage for:
int abs(int x) {
int ans = x;
if x < 0 {
ans = -x
}
return ans;
}test case: abs(-4)
test case: abs(4)
All branches taken in the program.
Path coverage v. Branch coverage?
Path coverage ensures each control flow path is taken.
Branch coverage ensures that each INDIVIDUAL branch is taken.
But there are combinations of branches that may not have been seen!
Condition Coverage
Test suite achieves path coverage if it results in each condition in each branching instruction being both true and false. (true, true) and (false, false) for if (a || b)
Combinatorial coverage
Test suite achieves combinatorial coverage if it results in each condition in each branching instruction being tried in all combinations of truth values.
What is required for combinatorial coverage:
if (avrageMark >= 50 or hardFailCount == 0 or elephantCount > 3) {
…
(50, 0, 4) = (true,true,true)
(50, 0, 3) = (true,true,false)
(50, 1, 4) = (true, false, true)
…
Modified Condition/Decision Coverage
100% branch coverage
100% condition coverage
each entry/exit point is exercised
each condition affects the behaviour independently.
if (a or b)
(TF), (FT), (FF)
Limitations of MC/DC
- For large number of inputs, A LOT OF TEST CASES.
- MC/DC is expensive and not a lot of evidence that it is valuable
- Used as a check box. Have you done it or not?
Source vs Object code coverage
Structural coverage at source level can differ from that at the object code level. Ie. multiple object code statements can be generated from a single source code statetement.
Structural coverage vs structural testing
SC: Statement, Decision, MC/DC
ST: Statement, branch, Condition, Path
SC: determine which code structure was not exercised by the requirements based test procedures.
ST: process of exercising software with test scenarios written from the source code not from requirements.
Structural testing limitations
provides no info about whether the code is doing what it’s supposed to be doing as specified in the requirements.
What is Criteria Subsumption
A Subsumes B if every test suite that satisfies A also satisfies B.
Branch coverage subsumes statement coverage.
If A subsumes B, will test suite satisfying A find all failures for B than a test suite satisfying B?
NO! in general this is true that test suites meeting subsuming criteria do find more failures but no guarantee.
Does test suite for branch coverage find all failures that a test suite satisfying statement coverage does?
int abs(int x) {
int ans = x;
if (x < 2) {
ans = -x
}
return ans;
}
test suite for branch coverage:
abs(10) = 10 pass
abs(-10) = 10 pass
achieves both statement and branch coverage.
test suite for statement coverage:
abs(-1) = -1 fail
