4. Test design techniques

Question 1

What should we know before executing a test?

Answer 1

Before we can actually execute a test, we need to know what we are trying to test, the inputs, the results
that should be produced by those inputs, and how we actually get ready for and run the tests.

Question 2

Where are test conditions, test cases and test procedures are documented?

Answer 2

Test Design specification, Test Case Specification and Test Procedure specification respectively

Question 3

What is test analysis?

Answer 3

Test analysis is the process of looking at something that can be used to derive
test information. This basis for the tests is called the ‘test basis’. It could be a
system requirement, a technical specification, the code itself (for structural
testing), or a business process. The test basis
includes whatever the tests are based on.

Question 4

What are test conditions?

Answer 4

From a testing perspective, we look at the test basis in order to see what could be tested - these are the test conditions. A test condition is simply something that we could test.

If we are looking to measure coverage of code
decisions (branches), then the test basis would be the code itself, and the list
of test conditions would be the decision outcomes (True and False).

Test conditions can be identified for test data as well as for test inputs and
test outcomes, for example, different types of record, different distribution of
types of record within a file or database, different sizes of records or fields in a
record.

Test conditions are documented in the IEEE 829 document called a Test
Design Specification

Question 5

How the experts can call ‘a list of

things that we could test’?

Answer 5

For example, Marick refers to ‘test requirements’ as
things that should be tested. Although it is not intended to imply that everything
must be tested, it is too easily interpreted in that way. [Marick, 1994] In con-
trast, Hutcheson talks about the ‘test inventory’ as a list of things that could be
tested [Hutcheson, 2003]; Craig talks about ‘test objectives’ as broad categories
of things to test and ‘test inventories’ as the actual list of things that need to be
tested [Craig, 2002].

Question 6

What is a strategy of test conditions identification?

Answer 6

When identifying test conditions, we want to ‘throw the net wide’ to identify
as many as we can, and then we will start being selective about which ones to
take forward to develop in more detail and combine into test cases. We could
call them ‘test possibilities’.
Having identified a list of test conditions, it is important to prioritize them,
so that the most important test conditions are identified (before a lot of time is
spent in designing test cases based on them). It is a good idea to try and think
of twice as many test conditions as you need

Question 7

What is a test technique?

Answer 7

A testing technique helps us select a good set of tests from the total number
of all possible tests for a given system. Different techniques offer different ways
of looking at the software under test, possibly challenging assumptions made
about it. Each technique provides a set of rules or guidelines for the tester to
follow in identifying test conditions and test cases.

Question 8

What is traceability?

Answer 8

Test conditions should be able to be linked back to their sources in the test
basis - this is called traceability. Traceability can be either horizontal through all the test documentation for
a given test level (e.g. system testing, from test conditions through test cases to
test scripts) or vertical through the layers of development documentation (e.g.
from requirements to components).

Question 9

Why traceability is important?

Answer 9

• The requirements for a given function or feature have changed. Some of the
fields now have different ranges that can be entered. Which tests were
looking at those boundaries? They now need to be changed. How many tests
will actually be affected by this change in the requirements? These questions
can be answered easily if the requirements can easily be traced to the tests.
• A set of tests that has run OK in the past has started to have serious prob
lems. What functionality do these tests actually exercise? Traceability
between the tests and the requirement being tested enables the functions or
features affected to be identified more easily.
• Before delivering a new release, we want to know whether or not we have
tested all of the specified requirements in the requirements specification. We
have the list of the tests that have passed - was every requirement tested?

Question 10

Should the detailed test cases be written down?

Answer 10

If you give an experienced user acceptance
tester with a strong business background a list of high-level test conditions, they
could probably do a good job of testing. But if you gave the same list to a new
starter who didn’t know the system at all, they would probably be lost, so they
would benefit from having more detailed test cases.

Test cases need to be detailed so that we can accurately check the results and
know that we have exactly the right response from the system. If tests are to be
automated, the testing tool needs to know exactly what to compare the system
output to.

Question 11

What is “kiddie testing?”

Answer 11

If we simply
put in some inputs and think ‘that was fun, I guess the system is probably OK
because it didn’t crash’, then are we actually testing it? We don’t think so. You
have observed that the system does what the system does - this is not a test.
Boris Beizer refers to this as ‘kiddie testing’ [Beizer, 1990]. We may not know
what the right answer is in detail every time, and we can still get some benefit
from this approach at times, but it isn’t really testing.

Question 12

What is an ‘oracle’?

Answer 12

In order to know what the system should do, we need to have a source of informa-
tion about the correct behavior of the system - this is called an ‘oracle’ or a test oracle.

Question 13

What the test case specifies?

Answer 13

In addition to the expected results, the test case also specifies the environ-
ment and other things that must be in place before the test can be run (the pre-
conditions) and any things that should apply after the test completes (the
postconditions).
The test case should also say why it exists - i.e. the objective of the test it is
part of or the test conditions that it is exercising (traceability).

Question 14

What is a test procedure or a test script?

Answer 14

The document that describes the steps to be taken in running a set of
tests (and specifies the executable order of the tests) is called a test proce-
dure in IEEE 829, and is often also referred to as a test script. Writing the test procedure is another opportunity to prioritize the tests, to
ensure that the best testing is done in the time available. A good rule of thumb
is ‘Find the scary stuff first’.

Question 15

What are the types or categories of test desing techniques from the perspective of their primary soruce?

Answer 15

A specification, the structure of the system or

component, or a person’s experience.

Question 16

What are the two main categories of testing techniques on a very general level?

Answer 16

Static and dynamic.
Dynamic techniques are subdivided into 
three more categories: specification-based (black-box, also known as behavioral techniques), structure-based (white-box or structural  techniques) and experi-
ence-based. Specification-based techniques include both functional and non-
functional techniques (i.e. quality characteristics).

Question 17

How else static testing techniques can be called?

Answer 17

non-execution techniques

Question 18

Where to apply the different categories of techniques?

Answer 18

Specification-based techniques are appropriate at all levels of testing (compo-
nent testing through to acceptance testing) where a specification exists.

Structure-based techniques can also be used at all levels of testing.
Developers use structure-based techniques in component testing and compo-
nent integration testing, especially where there is good tool support for code
coverage.

Experience-based techniques are used to complement specification-based
and structure-based techniques, and are also used when there is no specifica-
tion, or if the specification is inadequate or out of date.

Question 19

Provide examples of specification-based techniques

Answer 19

• equivalence partitioning;
• boundary value analysis;
• decision tables;
• state transition testing.

Question 20

Tell about Equivalence partitioning technique, please

Answer 20

Equivalence partitioning (EP) is a good all-round specification-based black-
box technique. It can be applied at any level of testing and is often a good tech-
nique to use first. It is a common sense approach to testing, so much so that
most testers practise it informally even though they may not realize it.

The idea behind the technique is to divide (i.e. to partition) a set of test con-
ditions into groups or sets that can be considered the same (i.e. the system
should handle them equivalently), hence ‘equivalence partitioning’.
Equivalence partitions are also known as equivalence classes - the two terms
mean exactly the same thing. The equivalence-partitioning technique then requires that we need test only
one condition from each partition.

Question 21

What is an invalid partition?

Answer 21

Note that when we say a partition is ‘invalid’, it doesn’t mean that it repre-
sents a value that cannot be entered by a user or a value that the user isn’t sup-
posed to enter. It just means that it is not one of the expected inputs for this
particular field. The software should correctly handle values from the invalid
partition, by replying with an error message such as ‘Balance must be at least
$0.00’.

Question 22

What is boundary value analysis?

Answer 22

Boundary value analysis (BVA) is based on testing at the boundaries between
partitions.

Question 23

What is an open boundary?

Answer 23

This is called an ‘open boundary’, because one of the sides of the partition is left
open, i.e. not defined.

Question 24

What is a boundary in respect of boundary value analysis?

Answer 24

With boundary value analysis, we think of the boundary as a dividing line
between two things. Hence we have a value on each side of the boundary (but
the boundary itself is not a value).

There is a school of thought that regards an actual value as a boundary
value. By tradition, these are the values in the valid partition (i.e. the values
specified). This approach then requires three values for every boundary two, so
you would have 0,1 and 2 for the left boundary, and 98, 99 and 100 for the right
boundary in this example. The boundary values are said to be ‘on and either
side of the boundary’ and the value that is ‘on’ the boundary is generally taken
to be in the valid partition.

Question 25

What is a good approach in designing a test case?

Answer 25

The more test conditions that can be covered in a single test case,
the fewer test cases will be needed in order to cover all the conditions. This is
usually the best approach to take for positive tests and for tests that you are rea-
sonably confident will pass. However if a test fails, then we need to find out why
it failed - which test condition was handled incorrectly? We need to get a good
balance between covering too many and too few test conditions in our tests.

Question 26

What is a safest option in designing test cases for invalid test conditions?

Answer 26

When we come to test invalid partitions, the safest option is probably to try
to cover only one invalid test condition per test case. This is because programs
may stop processing input as soon as they encounter the first problem.

However, if it is known that the software under test is required to process all
input regardless of its validity, then it is sensible to continue as before and
design test cases that cover as many invalid conditions in one go as possible.

Question 27

Why do both equivalence partitioning and boundary value analysis?

Answer 27

this approach may cause problems if that value fails - was it only the
boundary value that failed or did the whole partition fail?

Question 28

Tell about decision table (‘cause-effect’ table)

Answer 28

It is a good way to deal with combinations of things (e.g.
inputs). Decision tables provide a systematic way of stating complex business rules, which is useful for developers as well as for testers. Decision tables aid the systematic selection of effective test cases and can
have the beneficial side-effect of finding problems and ambiguities in the spec-
ification.

Question 29

What are other testing techniques that deal with testing combinations of things in addition to decision tables?

Answer 29

pairwise testing and orthogonal arrays

Question 30

When state transition testing is used?

Answer 30

is used where some aspect of the system can be
described in what is called a ‘finite state machine’. This simply means that the
system can be in a (finite) number of different states, and the transitions from
one state to another are determined by the rules of the ‘machine’. Any system where you get
a different output for the same input, depending on what has happened before,
is a finite state system.

Question 31

What are the basic part os state transition model?

Answer 31

• the states that the software may occupy (open/closed or funded/insufficient
funds);
• the transitions from one state to another (not all transitions are allowed);
• the events that cause a transition (closing a file or withdrawing money);
• the actions that result from a transition (an error message or being given
your cash).

Question 32

What is an advantage of state transition technique?

Answer 32

the model can
be as detailed or as abstract as you need it to be. Where a part of the system is
more important (that is, requires more testing) a greater depth of detail can be
modeled. Where the system is less important (requires less testing), the model
can use a single state to signify what would otherwise be a series of different
states.

Question 33

What is a state table?

Answer 33

In order to see the total number of
combinations of states and transitions, both valid and invalid, a state table is
useful.
The state table lists all the states down one side of the table and all the events
that cause transitions along the top (or vice versa). Each cell then represents a
state-event pair. The content of each cell indicates which state the system will
move to, when the corresponding event occurs while in the associated state.

Question 34

What is a use case testing?

Answer 34

Use case testing is a technique that helps us identify test cases that exercise the
whole system on a transaction by transaction basis from start to finish. A use case is a description of a particular use of the system by an actor (a
user of the system). Actors are generally people but they may also be other
systems. Use cases are a sequence of steps that describe the interactions
between the actor and the system. They serve as the foundation for developing test cases mostly at the system
and acceptance testing levels.

Question 35

Tell about structure based techniques

Answer 35

Structure-based techniques serve two purposes: test coverage measurement and structural test case design. They are often used first to assess the amount of testing performed by tests derived from specification-based techniques, i.e. to assess coverage. They are then used to design additional tests with the aim of increasing the test coverage.

Structure-based test design techniques are a good way of
generating additional test cases that are different from existing
tests.

Question 36

What is test coverage?

Answer 36

Test coverage measures in some specific way the amount of
testing performed by a set of tests (derived in some other way,
e.g. using specification-based techniques). Wherever we can count things and can tell whether or not each of those things has been
tested by some test, then we can measure coverage.

Coverage = (number of coverage items exercised/ total number of coverage items) * 100%
‘coverage item’ is whatever we have been able to count and see
whether a test has exercised or used this item.

100% coverage does not mean
100% tested!

One drawback of code coverage measurement is that it measures coverage of what has been written, i.e. the code itself; it cannot say anything about the
software that has not been written.

Question 37

At what levels coverage can be measured?

Answer 37

Component-testing level, integration-testing level or at system- or acceptance-testing levels.

at system or acceptance level, the coverage items may be require-ments, menu options, screens, or typical business transactions.

At integration level, we could measure coverage of interfaces or specific
interactions that have been tested.

Question 38

How coverage for each of the specification-based techniques can be measured?

Answer 38

• EP: percentage of equivalence partitions exercised (we could measure valid
and invalid partition coverage separately if this makes sense);
• BVA: percentage of boundaries exercised (we could also separate valid and
invalid boundaries if we wished);
• Decision tables: percentage of business rules or decision table columns
tested;
• State transition testing: there are a number of possible coverage measures:

• Percentage of states visited
• Percentage of (valid) transitions exercised (this is known as Chow’s 0-
  switch coverage)
• Percentage of pairs of valid transitions exercised (‘transition pairs’ or
  Chow’s 1-switch coverage) - and longer series of transitions, such as tran
  sition triples, quadruples, etc.
• Percentage of invalid transitions exercised (from the state table)

Question 39

What does coverage mean when it is discussed by business analysts, system testers or users?

Answer 39

it
most likely refers to the percentage of requirements that have been tested by a
set of tests. This may be measured by a tool such as a requirements manage-
ment tool or a test management tool.

Question 40

What does coverage mean when it is discussed by programmers

Answer 40

it most likely refers
to the coverage of code, where the structural elements can be identified using a
tool, since there is good tool support for measuring code coverage.

Question 41

How code coverage measurement tools can help to increase quality and productivity of testing?

Answer 41

They increase quality by
ensuring that more structural aspects are tested, so defects on those structural
paths can be found. They increase productivity and efficiency by highlighting
tests that may be redundant, i.e. testing the same structure as other tests

Question 42

What code coverage techniques are best to use on?

Answer 42

In common with all structure-based testing techniques, code coverage tech-
niques are best used on areas of software code where more thorough testing is
required.

Question 43

Tell about statement coverage, please

Answer 43

Statement coverage = (Number of statements exercised/total number of statements) * 100%
Different tools and
methods may count different things as statements, but the basic principle is the
same however they are counted. A statement may be on a single line, or it may
be spread over several lines.

Question 44

What is a decision?

Answer 44

A decision is an IF statement, a loop control statement (e.g. DO-WHILE or
REPEAT-UNTIL), or a CASE statement, where there are two or more possi-
ble exits or outcomes from the statement.

Question 45

How decision coverage is calculated?

Answer 45

Decision coverage = (nymber of decision outcomes exercised/ total number of decsion outcomes) * 100%

Question 46

How decision overage relates to statement coverage?

Answer 46

Decision coverage is stronger than statement coverage. It ‘sub-
sumes’ statement coverage - this means that 100% decision coverage always
guarantees 100% statement coverage.

So one test
is sufficient for 100% statement coverage, but two tests are needed for 100%
decision coverage.

Question 47

What is branch coverage technique?

Answer 47

branch coverage is closely related to decision coverage and at 100% coverage they give exactly the same results. Decision coverage measures
the coverage of conditional branches; branch coverage measures the coverage
of both conditional and unconditional branches.

Question 48

What are other control-flow code coverage measures?

Answer 48

linear code sequence and jump (LCSAJ) coverage, condition coverage, multiple condition coverage (also known as condition combination coverage) and condition determination
coverage (also known as multiple condition decision coverage or modified condition decision coverage, MCDC).

Question 49

How structure based techniques can also be called?

Answer 49

path testing
it is more correct to talk about
‘independent path segment coverage’

Question 50

What is exploratory testing?

Answer 50

It is a hands-on approach in which testers are involved in
minimum planning and maximum test execution. The planning involves the creation of a test charter, a short declaration of the scope of a short (1 to 2 hour)
time-boxed test effort, the objectives and possible approaches to be used.

A key aspect of exploratory testing is learning: learning by the tester about
the software, its use, its strengths and its weaknesses. As its name implies,
exploratory testing is about exploring, finding out about the software, what it does,
what it doesn’t do, what works and what doesn’t work.

This is an approach that is most useful when there are no or poor specifications and when time is severely limited.

Question 51

On which internal factors choosing a testing technique may depend?

Answer 51

• Models used - Since testing techniques are based on models, the models
available (i.e. developed and used during the specification, design and
implementation of the system) will to some extent govern which testing
techniques can be used. For example, if the specification contains a state
transition diagram, state transition testing would be a good technique to
use.
• Tester knowledge I experience - How much testers know about the system and
about testing techniques will clearly influence their choice of testing tech
niques. This knowledge will in itself be influenced by their experience of
testing and of the system under test.
• Likely defects - Knowledge of the likely defects will be very helpful in choos
ing testing techniques (since each technique is good at finding a particular
type of defect). This knowledge could be gained through experience of
testing a previous version of the system and previous levels of testing on the
current version.
• Test objective - If the test objective is simply to gain confidence that the soft
ware will cope with typical operational tasks then use cases would be a sen
sible approach. If the objective is for very thorough testing then more
rigorous and detailed techniques (including structure-based techniques)
should be chosen.
• Documentation - Whether or not documentation (e.g. a requirements
specification) exists and whether or not it is up to date will affect the
choice of testing techniques. The content and style of the documentation
will also influence the choice of techniques (for example, if decision tables
or state graphs have been used then the associated test techniques should
be used).
• Life cycle model - A sequential life cycle model will lend itself to the use of
more formal techniques whereas an iterative life cycle model may be better
suited to using an exploratory testing approach.

Question 52

On which external factors choosing a testing technique may depend?

Answer 52

• Risk - The greater the risk (e.g. safety-critical systems), the greater the need
for more thorough and more formal testing. Commercial risk may be influ
enced by quality issues (so more thorough testing would be appropriate) or
by time-to-market issues (so exploratory testing would be a more appropri
ate choice).
• Customer I contractual requirements - Sometimes contracts specify par
ticular testing techniques to use (most commonly statement or branch
coverage).
• Type of system - The type of system (e.g. embedded, graphical, financial, etc.)
will influence the choice of techniques. For example, a financial application
involving many calculations would benefit from boundary value analysis
• Regulatory requirements - Some industries have regulatory standards or
guidelines that govern the testing techniques used. For example, the aircraft
industry requires the use of equivalence partitioning, boundary value analy
sis and state transition testing for high integrity systems together with state
ment, decision or modified condition decision coverage depending on the
level of software integrity required.
• Time and budget - Ultimately how much time there is available will always
affect the choice of testing techniques. When more time is available we can
afford to select more techniques and when time is severely limited we will be
limited to those that we know have a good chance of helping us find just the
most important defects.

Question 53

What is a test case?

Answer 53

A set of input values, execution preconditions, expected results and execution
postconditions, developed for a particular objective or test condition, such as to exercise a
particular program path or to verify compliance with a specific requirement. [After IEEE 610]

Question 54

What is test case specification?

Answer 54

A document specifying a set of test cases (objective, inputs, test actions, expected results, and execution preconditions) for a test item. [After IEEE 829]

Question 55

What is a test procedure specification?

Answer 55

A document specifying a sequence of actions for the execution of a test. Also known as test script or manual test script. [After IEEE 829]

Question 56

What is black-box test design technique?

Answer 56

Procedure to derive and/or select test cases based on an
analysis of the specification, either functional or non-functional, of a component or system without reference to its internal structure.

Question 57

What is white-box test design technique?

Answer 57

Procedure to derive and/or select test cases based on an

analysis of the internal structure of a component or system.

Question 58

What is code coverage?

Answer 58

An analysis method that determines which parts of the software have been executed
(covered) by the test suite and which parts have not been executed, e.g. statement coverage, decision coverage or condition coverage.

Question 59

What is a decision coverage?

Answer 59

The percentage of decision outcomes that have been exercised by a test suite.

Question 60

What is a statement coverage?

Answer 60

The percentage of executable statements that have been exercised by a test
suite.

Question 61

How else white-box testing can be called?

Answer 61

structural testing,

structure-based testing

Question 62

What is error guessing?

Answer 62

A test design technique where the experience of the tester is used to anticipate what defects might be present in the component or system under test as a result of
errors made, and to design tests specifically to expose them.