Part IV Testing Flashcards
What is a test case specification?
A document describing the test case of a particular test
Contains:
a) Test case specification identifier
b) Test items
c) Input specifications
d) Output specifications
e) Environmental needs
f) Special procedural requirements
g) Intercase dependencies
What does a test procedure specification contain?
a) Test procedure specification identifier
b) Purpose
c) Special requirements
d) Procedure steps
What are XUnit family testing frameworks?
Frameworks that make specifying tests, executing tests, and reporting the results easier -> by automation
Why is it important to automate tests?
Manually running test scripts is tedious, error prone, and expensive
What is the difference between JUnit 3.8 and JUnit 4?
JUnit4 uses annotations (e.g. @Test) to identify test cases rather than reflection in JUnit 3.8
JUnit4 offers support for pre-test (set up) and post-test (tear down) management
How do you describe a test?
What is the IUT (Implementation under test)
What is the pretest state
What are the inputs (test case values and other inputs)
What is the expected state
What are some problems with test automation?
Certain artifact testing cannot be automated (testing user interfaces)
Hard to exercise potential failure situations (e.g. timing problems in real-time or concurrent software)
Some designs cannot be easily automatically tested
A lot of work needed to support traceability - relationship between tests executed and client’s requirements
What defines a good test?
The purpose of testing is to detect the presence of faults
Hence, the higher the probability of causing a failure (assuming a fault exists), the better the quality of the test
The quality of the overall test suite is more important than the quality of the individual tests
What is statement coverage?
The number of statements executed by all tests divided by total number of statements
The intuition is that each line may contain a fault and so the more lines that are executed the higher chance of causing a failure
It is a measure of test suite quality
Is 100% statement coverage always possible?
No, code may contain dead code (unreachable code)
What is white box testing?
Performing tests based on how the component is implemented (focus on the artifact e.g. code)
Decide on tests based on the code itself
Not good at detecting faults related to requirements
What is black box testing?
Performing tests based on what the component is supposed to do (focus on quality attribute)
Decide on tests based on requirements independent of the code
Not good at detecting faults in implementation decisions
What is a control flow graph?
DIrected graph used to model the control flow of a program
Vertex = statement
Edge = (A,B) if control flows from statement A to B
We are interested in the number of different ways control flow can occur through the code
What is a basic block?
A sequence of statements that have the following properties:
First statement is always the first statement executed (one entry point)
Only last statement can cause the program to begin executing code in a different basic block (one exit point)
ie. If first statement is executed then all statements in the basic block are executed
Can be used to simply control flow graphs
What is a test requirement?
A specific element of a software artifact that a test case must satisfy
e.g. statement A must be executed
What is a coverage criterion?
A rule or set of rules that impose test requirements on a test suite
e.g. the statement executed criterion imposes, for every statement x, the test requirement that statement x must be executed
What is coverage level?
The ratio of the number of test requirements satisfied by the test suite to the total number of test requirements
What is a test path?
Sequence of vertices in the CFG such that the first is the entry vertex, the last is the exit vertex and every pair of adjacent vertices is connected by an edge in the CFG
ie potential path through code
What is path coverage?
Coverage criterion - any test path
Intuition - any execution flow is represented by a path, so any missed paths may correspond to incorrect execution
What is condition coverage?
Given a compound expression (e.g. if ( a > 0 || b > 0) ) in order to detect all possible faults, all combination of sub-expressions must be tested
Can be modelled by breaking the compound expression into more if statements
What is the difference between a failure and an error according to JUnit?
Failure -> the expected result specified by human tester was not what JUnit observed (typically a failed assertion)
Error -> JUnit was unable to complete the test. Typically some exception that is thrown and not caught
What is a path vs test path?
A path is a CFG sequence of vertices
A test path is a path where the first vertex is the entry vertex and the last vertex is the exit vertex
What is a simple path?
Every vertex only appears once in the path
Or
The only vertex that appears twice is the first and last vertex in the path
What is a prime path?
A maximal length simple path
Path p is prime if it is simple (ie each vertex only appears once in the path) and has the maximum length (not a subpath of any other path)
What are the actions that can occur to data objects?
Defined (d) - data value comes into existence, assigned to variable
Killed or undefined (k) - data value no longer available
Usage (u) / Computation (c) - used for computation
Usage (u) / Predicate (p) - used in a predicate
How do you know a data object has been defined?
A location where a value is stored into memory
- variable appears on the left side of assignment operator
- formal parameter of a method (occurs in entry vertex)
- input to a program
- actual parameter in a method invocation and the method changes its value
How do you know a variable is being used?
A location where a variable’s value is accessed
- appears on right hand side of an assignment
- appears in a predicate (conditional test)
- an actual parameter in a method invocation
- return statement for a method
- output of a program
Data action order check: dd
(mostly) harmless, but worth checking
Data action order check: dk
possible fault
Data action order check: du
normal case
Data action order check: kd
normal case
Data action order check: kk
it’s dead already (suspicious)
Data action order check: ku
fault (not possible in some languages)
Data action order check: ud
probably not a fault
Data action order check: uk
normal
Data action order check: uu
normal
Data action order check: -k
variable not defined but then killed
Data action order check: -d
first definition in path -> normal
Data action order check: -u
used before defined
Issues with global data
Data action order check: k-
killed is the last thing that happens
Data action order check: d-
defined but never used
Issues with global data
Data action order check: u-
used but never killed, probably ok
What is a DU-path?
Checking whether a value that is defined is correct for a given use
A du-path with respect to variable x is a simple path (series of vertices) that is def-clear with respect to x such that the first vertex of the path is where x is defined and the last vertex is where x is used
What is all du-paths?
Visit every du path for every variable
- strongest data flow strategy
- not as strong as all paths
Are du-paths practical?
All du-paths requires 2^t in the worst case, where t is the number of conditional transfers (conditions)
All du-paths may not produce a better test suite (or even more tests) than branch coverage, but it usually does better
What is the procedure for creating a test suite based on du-paths?
Identify definitions of values
Identify uses of those definitions
Identify (control flow) paths from definitions of values to their uses (du-paths)
Choose a set of du-paths based on chosen testing strategy (e.g. all paths, all predicate-use)
Create a test suite that causes all of the chosen du-paths to be followed
What is equivalence partitioning?
The idea that you should be testing inputs that add value to the test -> increase the chances of failure
Partition possible input values into equivalence classes such that every possible input value is in at most one equivalence class (classes are disjoint) and every possible input value is in at least one equivalence class (union of all classes is all possible input values)
The aim should be that there is at least one input value tested from each equivalence class
What is boundary value analysis?
Choosing test cases from the boundaries of the equivalence classes as faults often occur at the edges
Should aim to have inputs from both sides of every boundary
What is a predicate?
An expression that evaluates to a boolean value
Predicates can contain:
- boolean variables
- relationship expressions
- methods that return booleans
- expressions involving logical operators (!, &&, ||, etc)
What is a clause?
A predicate with no logical operators - not, and, or, etc
What is predicate coverage?
Given a predicate p, there are two test requirements - p evaluates to true, and p evaluates to false
Same as branch coverage
What is clause coverage?
For every clause in a predicate p, there are two test requirements - c evaluates to true or false
What is combinatorial coverage?
For a predicate p, there is a test requirement for every combination of truth values for every clause c in p
Similar to condition coverage
What are the advantages and disadvantages of combinatorial coverage?
Predicate coverage does not exercise all clauses
Clause coverage does not always ensure predicate coverage
However combinatorial coverage grows exponentially -> subset of combinatorial coverage that tests each clause independently from the other clauses
Conclusion -> find a good subset of combinatorial coverage such that still gives a good quality test suite
What is an active clause?
For a given predicate, choose one clause as the major one -> rest become minor clauses
An active clause is a major clause that will determine the value of the predicate
What is active clause coverage?
For each predicate choose major clauses and ensure that all major clauses evaluate to true and false.
Active clause coverage maximises the variation of the value of the predicate white restricting the number of variation of clauses needed ie only change clauses that matter
