Combinatorial Testing

Question 1

What is the idea behind Combinatorial Testing?

Answer 1

To identify the distinct attributes that can be varied - data, environment, configuration.. And systematically generate combinations to be tested. Idea is that tests are varied and include possible edge cases.

Question 2

Main three ideas of combinatorial approaches?

Answer 2

1. Category partition testing, 2. Pairwise testing,

3. Catalogue based testing

Question 3

What is meant by category-partition testing?

Answer 3

To separate, manual, identification of values that characterise input space from automatic generations of combinations of test cases.

Question 4

What is meant by pairwise testing?

Answer 4

To systematically test interactions among attributes of the program input space with a small number of test cases

Question 5

What is meant by catalog-based testing?

Answer 5

To aggregate and synthesise experience of test designers in an organisation or domain, to aid in identifying attribute values.

Question 6

Steps for Category Partition?

Answer 6

1. Decompose spec into independently tested features. For each feature identify parameters and env elements, for each pair of these identify elementary category/characteristics.
2. Identify relevant values for each characteristic (normal, boundary, special, error values).
3. Introduce constraints.

Question 7

When the specification is informal, what is a way to approach testing?

Answer 7

To check the configuration of the system - the validity of a computer configuration, the parameters for this check being the model and the set of components.

Question 8

What is the model parameter in checking the configuration of a system?

Answer 8

The model identifies a product and the set of constraints on the components. Models are characterised by the logical slots for components, which may be required or optional. Required slots must be given suitable component for legality while optionals are filled in as necessary.

Question 9

What is meant by set of components as a parameter in checking the configuration of a system?

Answer 9

A set of (slot, component) pairs corresponding to the required and optional slots for a model. Components are a choice to vary and is not designed to be replaced by the end user. The value empty is allowed for optional slots. In addition to compatability between models and slots, individual components may in/compatible with each other.

Question 10

First step to combinatorial testing?

Answer 10

To identify independently testable units and categories.

Question 11

How to identify independently testable categories?

Answer 11

Choosing categories: no specific rules, not easy. They should reflect designer’s judgement regarding what values should be treated differently. Requires experience and knowledge of the application domain and product architecture. Ideally identify hidden characteristics.

Question 12

How to identify independently testable units?

Answer 12

Model (parameter) - number, number of required slots, number of optional slots. Components (parameter) - correspondence of selection with model slots, required component selection… etc. Product database (environment element) - number of models/components in database.

Question 13

How to identify relevant values in the process of CT?

Answer 13

Identify representative classes of values for each category. These can be identified by applying 1. Boundary value testing or 2. Erroneous condition testing

Question 14

What is meant by Boundary value testing?

Answer 14

Selecting extreme values within a class, values outside but close to a class, interior non-extreme values of the class …

Question 15

What is meant by Erroneous condition testing?

Answer 15

Selecting values outside the normal domain of the program.

Question 16

What is the step after identifying relevant values in CT?

Answer 16

Introducing constraints! To rule out impossible combinations,and reduce the size of the test suite if too large.

Question 17

What is the error constraint in CT?

Answer 17

Errors indicate value classes that correspond to erroneous values AND need to be tried only once. There is no need to test all possible combinations of errors, sometimes just one as it implies certain logic.

Question 18

How are property constraints done in CT?

Answer 18

constraint PROPERTY (IF PROPERTY) where property= groups values of single parameter to identify subsets of values with common properties AND if property= bounds the choices of values for a category that can be combined with a particular value from a different category.

Question 19

What are single constraints in CT?

Answer 19

Single indicates a value class that test designers chose to test only once to reduce the number of test cases.

Question 20

Difference between single and error processes in CT?

Answer 20

Single and error have the same effect but differ in rationale. Keeping them distinct is important for documentation and regression testing.

Question 21

What does category partition achieve? Downside?

Answer 21

A systematic approach - identifies characteristics and values, and generates combinations. Works well when intuitive constraints reduce number of combinations to a few test cases. However, test suite grows rapidly with the number of categories.

Question 22

What does pairwise combinatorial testing achieve?

Answer 22

A non exhaustive approach to category partition. Combines values systematically but not exhaustively. The idea is that most unplanned interactions are among few parameters characteristics. So pairwise combination generates combinations that efficiently cover other classes, revealing most faults.

Question 23

What does catalog based testing achieve?

Answer 23

An improvement on manual step of deriving value classes. Gathering experience in systematic collection this approach can speed up test design, routinise many decisions. The catalogues capture designers’ experience by listing important cases for each variable type.

Question 24

What are the steps to catalog based testing?

Answer 24

1. Analyse initial spec to identify simple elements - ops, vars.. 2. Derive first set of test case specs from pre/post conditions and definitions. 3. Complete set of test case specs using test catalogues.

Question 25

What is cgi_decode?

Answer 25

A function that translates a cgi-encoded string to plain ASCII, reversing encoding applied by the Common Gateway Interface of most servers. CGI makes spaces to +
and non letters to hex seqs. cgi_decode maps all this to ascii and returns a 0/1 for success.

Question 26

Difference between validated and assumed preconditions?

Answer 26

Validated - checked by system, Assumed - assumed by system.

Question 27

What elements does cgi_encode identify?

Answer 27

Preconditions, postconditions, variables, operations and definitions.

Question 28

How is the second step of catalog based testing achieved? (deriving initial set of test case specs)

Answer 28

Using cgi_decode to create tests from the preconditions, tests for the postconditions, and tests from variables.

Question 29

How is the catalog applied in catalog based testing?

Answer 29

1. Scan code sequentially
2. For each element in catalog scan the specs, and apply catalog entry
3. Delete redundant test cases

Question 30

What is a catalog in catalog based testing?

Answer 30

A list of kinds of elements that can occur in a specification. Each catalog entry is associated with a list of generic test case specs.

Question 31

What is generally an obstacle to automatic analysis of specifications?

Answer 31

The flexibility and expressiveness of natural language.

Question 32

What are the main two aspects of combinatorial testing?

Answer 32

1. A manual step of structuring specs into sets of properties. 2. Automatizable step of producing valuable combinations of these choices.