Chapter 2 - White-Box Test Techniques Flashcards
Certification: 8 questions worth 17 points total.
True or False: White-box test techniques apply to code and other structures with a control flow, such as business process flow charts. Achieving full coverage (i.e. 100%) means that the entire set of tests is complete.
False.
Each specific technique enables test cases to be derived systematically and focuses on a particular aspect of the structure. The test cases generated by the techniques satisfy coverage criteria which are set as an objective and are measured against.
Achieving full coverage (i.e. 100%) does not mean that the entire set of tests is complete, but rather that the technique being used no longer suggests any useful further tests for the structure under consideration.
True or False: Test inputs are generated to ensure a test case exercises a particular part of the code (e.g., a statement or decision outcome). Determining the test inputs that will cause a particular part of the code to be exercised is simple, given access to the requirements or design specification.
False.
Determining the test inputs that will cause a particular part of the code to be exercised can be challenging, especially if the part of the code to be exercised is at the end of a long control flow sub-path with several decisions on it. The expected results are identified based on a source external to this structure, such as a requirements or design specification or another test basis.
True or False: Achieving full statement coverage should be considered as a minimum for all code being tested.
True, although this is not always possible in practice due to constraints on the available time and/or effort.
True or False: High percentages of statement coverage will detect all defects in the code’s logic.
False.
Even high percentages of statement coverage may not detect certain defects in the code’s logic. In many cases achieving 100% statement coverage is not possible due to unreachable code. Although unreachable code is generally not considered good programming practice, it may occur, for instance, if a switch statement must have a default case, but all possible cases are handled explicitly.
True or False: Functionally, the two below categories of testing are distinct.
Decision testing exercises the decision outcomes in the code. It considers the entire decision as a whole, and evaluates each TRUE or FALSE outcome.
Branch testing exercises the branches in the code, where a branch is normally considered to be an edge of the control flow graph.
False.
Branch testing is often used interchangeably with decision testing, because covering all branches and covering all decision outcomes can be achieved with the same tests.
For programs with no decisions, the definition of decision coverage results in a coverage of 0/0, which is undefined, no matter how many tests are run, while the single branch from entry to exit point (assuming one entry and exit point) will result in 100% branch coverage being achieved. How do we reconcile these measures, given that branch and decision testing are functionally the same?
To address this difference between the two measures, ISO 29119-4 requires at least one test to be run on code with no decisions to achieve 100% decision coverage, so making 100% decision coverage and 100% branch coverage equivalent for nearly all programs.
Essentially, we treat code with 0 decisions as having 1 decision (made by the programmer ahead of time), and thus 1 test provides full decision coverage.
True or False: In situations where a decision with multiple conditions is made, Decision Testing will always detect defects caused by specific combinations of condition outcomes.
False.
Decision testing does not consider the details of how a decision with multiple conditions is made and may fail to detect defects caused by combinations of the condition outcomes.
When multiple atomic conditions can affect the outcome of a decision in code, we may turn to Modified Condition/Decision Testing. Assuming N unique, mutually independent atomic conditions, how many times must we exercise the decision to achieve MC/DC coverage?
N+1 times.
For example, given 3 independent atomic conditions affecting a decision, each of which resolves to a Boolean value, the possible combinations under test are:
TRUE - TRUE - TRUE
FALSE - TRUE - TRUE
TRUE - FALSE - TRUE
TRUE - TRUE - FALSE
This technique checks that each of the atomic conditions independently and correctly affects the outcome of the overall decision. Note that we don’t need to cover all possible combinations, we merely need to validate that each condition independently affects the outcome as expected.
Which of the following text techniques provides a reasonable middle ground of rigorous testing without requiring extremely high volumes of test cases, when testing safety critical software systems?
- Decision Testing. This level of coverage should be considered when the code being tested is important or even critical.
- Modified Condition/Decision Testing. It is used when testing software where a failure may cause a catastrophe.
- Multiple Condition Testing. This technique is used to test high risk software and embedded software which are expected to run reliably without crashing for long periods of time.
Modified Condition/Decision Testing.
This technique is used in the aerospace and automotive industries, and other industry sectors for safety-critical systems.
Modified condition/decision testing can be a
reasonable middle ground between decision testing and multiple condition testing (due to the large number of combinations to test). It is more rigorous than decision testing but requires far fewer test conditions to be exercised than multiple condition testing when there are several atomic conditions in the decision.
Achieving MC/DC may be complicated when there are multiple occurrences of the same variable in a decision with multiple conditions; when this occurs, the conditions may be “coupled”. Depending on the decision, it may not be possible to vary the value of one condition such that it alone causes the decision outcome to change.
How can we resolve this?
One approach to addressing this issue is to specify that only uncoupled atomic conditions are tested using modified condition/decision testing .
The other approach is to independently analyze each decision in which coupling occurs.
In rare instances, it might be required to test all possible combinations of atomic conditions that a decision may contain. This level of testing is called multiple condition testing.
Assuming N unique, mutually independent atomic conditions, how many times must we exercise a decision to achieve full multiple condition coverage for it?
Assuming N unique, mutually independent atomic conditions, full multiple condition coverage for a decision can be achieved by exercising it 2^N times. Note that a single test case may exercise several condition combinations and therefore it is not always necessary to run 2^N separate test cases to achieve 100% multiple condition coverage.
For example, given 3 independent atomic conditions, the possible combinations are:
TRUE - TRUE - TRUE
TRUE - FALSE - TRUE
TRUE - TRUE - FALSE
TRUE - FALSE - FALSE
FALSE - TRUE - TRUE
FALSE - FALSE - TRUE
FALSE - TRUE - FALSE
FALSE - FALSE - FALSE
2^3 = 8, which confirms that we have found all possible combinations.
Which of the following is not true of Multiple Condition Testing?
- The total number of combinations can be derived directly from a truth table containing all the atomic conditions, making all possible test cases easily determined.
- The sheer number of test cases required for Multiple Condition Testing makes it infeasible in most situations involving several atomic conditions in a decision.
- Compilers using “short circuiting” optimizations to skip expressions where the final outcome can be determined do not pose an issue for Multiple Condition Testing.
Short Circuiting does pose an issue for Multiple Condition Testing.
Some compilers and/or interpreters are designed such that they exhibit short-circuiting behavior when evaluating a complex decision statement in the code. That is, the executing code may not evaluate an entire expression if the final outcome of the evaluation can be determined after evaluating only a portion of the expression. For example, if evaluating
the decision “A and B”, there is no reason to evaluate B if A has already been evaluated as FALSE. No value of B can change the final result, so the code may save execution time by not evaluating B.
If the compiler uses short-circuiting, the number of condition combinations that can be exercised will often be reduced, depending on the order and grouping of logical operations that are performed on the atomic conditions.
True or False: API Testing is a type of testing rather than a technique.
True.
An application programming interface (API) is a defined interface that enables a program to call another software system, which provides it with a service, such as access to a remote resource. Typical services include web services, enterprise service buses, databases, mainframes, and Web UIs.
Which is more important in API testing?
- Negative testing. Programmers that use APIs to access services external to their own code may try to use API interfaces in ways for which they were not intended. That means that robust error handling is essential to avoid incorrect operation.
- Positive, or “golden path” testing. The focus is on the evaluation of input values and returned data.
Negative testing is often crucial when dealing with APIs.
Programmers that use APIs to access services external to their own code may try to use API interfaces in ways for which they were not intended. That means that robust error handling is essential to avoid incorrect operation.
Combinatorial testing of many interfaces may be required because APIs are often used in conjunction with other APIs, and because a single interface may contain several parameters, where values of these may be combined in many ways.
What API architectural style is the de-facto standard for software integration?
RESTful (Representational State Transfer) Web services allow requesting systems to access Web resources using a uniform set of stateless operations.
Several coverage criteria exist for RESTful web APIs. They can be divided into two groups: input coverage criteria and output coverage criteria. Describe each of these groups.
Among others, input criteria may require the execution of all possible API operations, the use of all possible API parameters, and the coverage of sequences of API operations.
Among others, output criteria may require the generation of all correct and erroneous status codes, and the generation of responses containing resources exhibiting all properties (or all property types).
Which of the following is not true of API testing?
- An organization that provides an API interface must ensure that all services have a very high availability; this often requires strict reliability testing by the API publisher as well as infrastructure support.
- APIs rarely are loosely coupled, making lost transactions or timing glitches unlikely. This reduces the need for thorough testing of the recovery and retry mechanisms.
- At a minimum, the API test should include making calls to the API with both realistic input values and unexpected inputs for checking exception handling.
- More thorough API tests may ensure that callable entities are exercised at least once, or all possible functions are called at least once.
APIs frequently are loosely coupled, resulting in the very real possibility of lost transactions or timing glitches. This necessitates thorough testing of the recovery and retry mechanisms.
When specifying a required white-box coverage level it is good practice to only specify it at 100%. Why should lower levels of coverage not be required?
The reason for this is that if lower levels of coverage are required, then this typically means that the parts of the code that are not exercised by testing are the parts that are the most difficult to test, and these parts are normally also the most complex and error-prone.
So, by requesting and achieving for example 80% coverage, it may mean that the code that includes the majority of the detectable defects is left untested.
When specifying the required white-box coverage for a test object, how many coverage criteria should be listed?
- All of them in use
- The most important ones, according to the test objectives
- Only one
It is also only necessary to specify this for a single coverage criterion (e.g., it is not necessary to require both 100% statement coverage and 100% MC/DC).
One coverage criterion is said to subsume another if, for all components and their specifications, every set of test cases that satisfies the first criterion also satisfies the second. For example, branch coverage
subsumes statement coverage because if branch coverage is achieved (to 100%), then 100% statement coverage is guaranteed.
At what percent is it best practice to specify white-box coverage levels?
100%.
The reason for this is that if lower levels of coverage are required, then this typically means that the parts of the code that are not exercised by testing are the parts that are the most difficult to test, and these parts are normally also the most complex and error-prone.
Why might different levels of coverage be defined in the requirements for different parts of a system?
Because different parts of a system contribute differently to risk. For instance, in an avionics system, the subsystems associated with inflight entertainment would be assigned a lower level of risk than those associated with flight control.
When selecting white box test techniques for non-safety-related systems, what do the following factors have in common?
- contract
- customer
- regulatory standards
- organizational test strategy
Not aligning with the expectations or requirements from these sources may lead to legal and business problems, including breach of contract, fines, and censure from management.
When selecting white box test techniques for non-safety-related systems, what do the following factors have in common?
- Historical Defect Information
- Tester Skills and Experience
- Test Tools
Failing to consider these factors fully can result in increased risk if you choose a white box test technique that doesn’t account for past issues, tester ability, and your available tools.
Where the software being tested is part of a safety-related system, then a regulatory standard which defines the required coverage levels to be achieved will normally have to be used. What must be performed to assess risks and assign integrity levels to each part of the system?
Such standards typically require a hazard analysis to be performed for the system and the resultant risks are used to assign integrity levels to different parts of the system. Levels of required coverage are defined for each of the integrity levels.
