CHAPTER 4 TEST ANALYSIS AND DESIGN Part 1 Flashcards
ACCEPTANCE CRITERIA
Criteria that a component or system must satisfy in order to be accepted by a user, customer or other authorized entity
BLACK-BOX TESTING TECHNIQUE
A test technique based on an analysis of the specification of a component or system
SPECIFICATION BASED TECHNIQUE
COVERAGE
The degree to which specified coverage items are exercised by a test suite, expressed as a percentage
COVERAGE ITEM
An attribute or combination of attributes derived from one or more test conditions by using a test technique
DECISION TABLE TESTING
A black-box testing technique in which test cases are designed to exercise the combinations of conditions and the resulting actions shown in a decision table
STATE TRANSITION TESTING
An approach testing in which the testers dynamically design and execute tests based on their knowledge, exploration of test item, and the results of previous testing
TEST TECHNIQUE
A procedure used to define test conditions, design test cases, and specify test data
TEST DESIGN TECHNIQUE
WHITE-BOX TEST TECHNIQUE
A test technique only based on the internal structure of a component or system
STRUCTURE-BASED TECHNIQUE
„ERROR(defect) HYPOTHESIS” IN TEST TECHNIQUES - ERROR/DEFECT THEORY
• scientific basis for the construction of each technique
• what types of problems this technique is capable of detecting
• each technique is built around an abstract problem related to a specific error hypothesis and designed to detect specific type of error/defect
3 CATHEGORIES OF TEST TECHNIQUES DIVIDED BY THE SOURCE OF KNOWLEDGE
- BLACK-BOX TECHNIQUES (4)
SOURCE: specification (external source of knowledge about the test object)
TEST BASIS: design documentation - requirements specification - use cases - business process - WHITE-BOX TECHNIQUES (2)
s: structure (internal architecture of the test object)
T.B. : code - menu structure - business process flow structure - architecture description - EXPERIENCE-BASED TECHNIQUES (3)
S: knowledge (experience, intuition, common sense)
BLACK-BOX TEST TECHNIQUES
Behavioural techniques
Specification-based techniques
• knowledge external to the test object about HOW it should BEHAVE
•requirements - use cases - user stories - business processes = DESCRIPTION OF DESIRED BEHAVIOUR OF THE SYSTEM
BOTH FUNCTIONAL AND NON-FUNCTIONAL WITHOUR REFERRING TO ITS INTERNAL DESIGN
ADVANTAGES OF BLACK-BOX TECHNIQUES
• documentation exists long before the implementation of a component or system - CREATING TESTS EARLY
• FOR STATIC TESTING, DYNAMIC
• BOTH FUNCTIONAL AND NONFUCNTIONAL
WHITE-BOX TESTING TECHNIQUES
Structure - structure-based techniques
• basis - INTERNAL STRUCTURE OF TEH TEST OBJECT
• this structure = code, high-level model of TEH system or module architecture - graph or information flow in business process
• CODE CAN NEVER BE AN ORACLE FOR ITSELF
EXPERIENCE-BASED TECHNIQUES
• „soft” sources of knowledge about the system under test - intuition, experience, knowledge of defetcs found in previous versions of
• referring to skills of a tester and other stakeholders
4 BLACK-BOX TESTING TECHNIQUES
- EQUIVALENCE PARTITIONING (EP)
- BOUNDARY VALUE ANALYSIS (BVA)
- DECISION TABLE TESTING
- STATE TRANSITION TESTITNG
2 WHITE-BOX TESTING TECHNIQUES
- STATEMENT TESTING AND STATEMENT COVERAGE
- BRANCH TESTING AND BRANCH COVERAGE
3 EXPERIENCE-BASED
- ERROR GUESSING
- EXPLORATORY TESTING
- CHECKLIST-BASED TESTING
FACTORS INFLUENCING CHOICE OF TEST TECHNIQUE
- FORMAL FACTORS
documentation - law and regulations - customer contract provisions - processes in place in the organization -test objectives - SDLC model) - PRODUCT FACTORS
software, its complexity - importance of various quality characteristics, risks - expected types of defects - expected use of the software - PROJECT FACTORS
available time - budget - resources - tools - skills - knowledge and experience of testers
EXAMPLES TEST TECHNIQUES WITH DIFFERENT LEVELS OF FORMALIZATION
- VERY LOW
unplanned, undocumented execution of error guessing, without saving test results - LOW
conducting exploratory tests with a test charter - MEDIUM
using decision table testing to test business logic; test cases are documented in test case specifications and test results are logged - LARGE
using the state machine model and state transition testing technique to test program behaviour
test design (state machine), test cases (in form of test scripts), and test results (logs) are documented
EQUIVALENCE PARTITIONING (EP)
• purpose - overcome the principle of impossibility of thorough testing
there are usually infinite number of possible inputs, but the number of various EXPECTED BEHAVIOURS of a program on these inputs is TYPICALLY FINITE if we consider specific behaviours related to a strictly defined aspect of app’s operation
• in this method
DIVISION OF A GIVEN DOMAIN INTO SUBSETS - PARTITIONS - SUCH AS FOR EVERY TWO ELEMENTS FROM ONE PARTITION, WE HAVE IDENTICAL PROGRAM BEHAVIOUR
- for example age under 18 certain behaviours - values belonging to a partition below 18 are treated the same way -> each element of a given partition is an equally good choice to test
APPLICATION OF EP
• any situation, any test level, any type of test - division of possible data into groups
•it can be applied not only to input domains but also to output and internal domains
• THE DOMAIN DOESN’T HAVE TO BE A NUMERICAL DOMAIN - ANY NONEMPTY SET, EXAMPLES:
- a set of natural numbers (e.g. partition in to even and odd numbers)
- a collection of words (e.g. partition by word length, one-letter, two-letter etc.)
- collections relating to time (e.g. partition by year of birth, by month ona given year etc)
- a collection of operating system types: {Windows, Linux, macOS}
PARTITIONING CORRECTNESS
• EACH ELEMTN OF THE DOMAIN BELONGS TO EXACTLY ONE EQUIVALENCE PARTITION
• NO EQUIVALENCE PARTITION IS EMPTY
VALILD PARTITIONS - partitions that contain „correct”/„normal” values, i.e. values accepted/expected by the system
OR - values that the specification defines its processing
INVALID PARTITIONS - partitions that contain values that the component or a system should reject (e.g. data with incorrect syntax, exceeding acceptable ranges)
OR - values for which the specification doesn’t defines the processing
DERIVING TEST CASES IN EP
COVERAGE ITEMS IN EP - equivalence partitions
• minimum set of test cases to ensure 100% coverage => one that covers every equivalence partition = for each identified partition, there is a test case containing a values from that partition
ONE-DIMENSIONAL CASE
one domain and one division - minimum number of cases - the number of equivalence partitions we identified
MULTIDIMENSIONAL CASE
more than one domain
• number of test cases depend on the way we treat combinations of invalid partitions AND on possible dependencies or constraints between values and partitions from different domains
COVERAGE - number of equivalence partitions tested using at least one value/total number of equivalence partitions partitions defined *%
COVERING MULTIPLE EQUIVALENCE PARTITIONS SIMULTANEOUSLY
DEFECT MASKING
Test must simultaneously cover equivalence partitions derived from more than one domain (both need to be True, risk of testing just one is valid, the other False, which may seem ok, but it’s not)
Steps
1. Create the smallest possible number of test cases composed only of test data from valid partitions, which will cover all valid partitions from all domain
2. For each uncovered invalid partition, create a separatete test case in which data from that partition will occur, and all other data will come from the valid partitions
„EACH CHOICE” COVERAGE
• in multidimensional case - more than one domain + each test covers one partition from the distribution of each domain
• simple but weak
• tester tries to make the next test case cover as many previously uncovered coverage items as possible
E.g 4 browsers (Chrome C, Firefox, Safari S, Opera O) and 3 OS (Windows, Linux, macOS)
In this method 4 tests
TC1: C, W
TC2: F, L
TC3: S,IOS
TC4: O,W
TYPES OF PROBLEMS DETECTED BY EP
• problems resulting from faulty data processing, resulting from errors in the domain model
DEFINING THE DOMAIN IS KEY
always applied to a specific domain that models the problem
NOT GREAT METHOD TO ANALYSE DYNAMIC DOMAINS - then state transitioning testing
GOOD AT DETECTING ISSUES IN THE IMPLEMENTATION OF THE STATIC DOMAINS
BOUNDARY VALUE ANALYSIS (BVA)
• built on the EP
• SELECTING VERY SPECIFIC ELEMENTS OF EQUIVALENCE PARTITIONS FOR TESTING - THOSE THAT LIE ON THE BOUNDARIES OF THESE PARTITIONS
• BOUNDARY VALUE OF A PARTITION - the smallest or the largest element of a partition, so there needs to be an order of elements in the domain
ONLY TO DOMAINS THAT ARE ORDERED, IN SOME ORDER RELATION
SETS OF NUMBERS, INTEGER OR REAL NUMBERS, VALUES RELATING TO TIME OR DATE
NO GAPS IN EQUIVALENCE PARTITIONING
DERIVING TEST CASES WITH BVA
- Identify the domain to analyse
- Perform equivalence partitioning of this domain into equivalence partions
- For each equivalence partition identified, determine boundary values
- For each boundary value determine the coverage items (the elements to be tested) for this boundary value determine
2-VALUE BVA VS 3-VALUE BVA
• 2-VALUE
for each identified boundary values, that value and its nearest neighbour NOT BELONGING are selected for testing
partition = {1,2,3,4,5,6}
tests: t1 = 1; t2=0;
T3 = 6, t4 = 7
• 3-VALUE
for each identified boundary value - we take both neighbours + that value, regardless of partition they belong
partition = {1,2,3,4,5,6}
TESTS: t1 = 1, t2 =0, t3 =2
T4 = 6, t5 = 5, t6 = 7
APPLICATION OF BVA
• very common mistakes made by developers: > instead of >=
VALUES OUTSIDE THE DOMAIN
Depends on the interface, if developer allowed in his code incorrect values/incorrect input, e.g. below 0 numbers
COVERAGE IN BVA
• in 2-VALUE
quotient of the number tested boundary values and the total number of identified boundary values
• 3-VALUE
quotient of the number tested with their neighbours and the total number of identified boundary values and their neighbours
DECISION TABLE TESTING - APPLICATION
• technique used to verify the correctness of BUSINESS RULES IMPLEMENTATION
• usually takes form of LOGICAL IMPLICATION
IF (condition/ combination of conditions) THEN (action)
where both condition and action can be composed of more than one factor
E.g. IF (customer_age < 18) THEN (assign a ticket discount)
• SYSTEMATICALLY TEST THE CORRECTNESS OF TEH IMPLEMENTATION OF COMBINATIONS OF CONDITIONS
COMBINATORIAL TECHNIQUES
CONSTRUCTION OF DECISION TABLE
• E.G. table consisting of 2 part: describing conditions and actions
CONDITIONS: 1. customer having a loyalty card; 2. Total amount of purchases above $1000 3. Purchase in the last 30 days
ACTIONS: assigning discounts: 0%, 5%, 10%
DERIVING TEST CASES FROM THE DECISION TABLE - STEPS
- Identify all possible single conditions (form test basis, e.g. based on specifications, customer conversations, common sense) and list them in consecutive rows in the upper part fo the table
- Identify all corresponding actions that can occur in the system depending on theses conditions (also derived from test basis), and list them in consecutive lines in lower part of the table
- Generate all combinations of conditions, eliminate infeasible combinations of conditions; for each feasible combination, a separate column is created with the values of each condition listed in this column
- Identify, for each identified combination of conditions, which actions nad how they should occur
- For each column of the decision table design a test cale in which the test input represents combination of conditions specified in this column; test is passed if, after execution, the system takes actions as described at the bottom part of the table
NOTATION AND POSSIBLE ENTIRES IN THE DECISION TABLE
• typically conditions and values take form of logical values - true or false, but they can be any object: numbers, ranges of numbers, category values, equivalence partition etc
• decision table with only Boolean - LIMITED-ENTRY DECISION TAVLE
• any other than Boolean values - EXTENDED-ENTRY DECISION TABLE
HOW TO DETERMINE ALL COMBINATIONS OF CONDITONS IN DECISION TABLE
• tree method to systematically determine all combinations of conditions
INFEASIBLE COMBINATIONS
•not feasible because of badly phrased partition
• illogical if tried all combinations
MINIMIZING THE DECISION TABLE
• spotting irrelevant values (e.g. combination of conditions - age and smokes) or payment type (cash or card) and correct pin => not applicable
• collapsing/minimizing decision table - RISKY, CAN HIDE A DEFECT - RISK MITIGATION EXERCISE
COVERAGE IN DECISION TABLES
COVERAGE ITEMS = INDIVIDUAL COLUMNS OF THE INDIVIDUAL COLUMNS OF THE TABLE CONTAINING POSSIBLE COMBINATIONS OF CONDITIONS (i.e. feasible columns))
COVERAGE counts against the number of feasible columns of the decision table, not all possible combinations
DECISION TABLES AS A STATIC TESTING TECHNIQUE
• great at detecting problems with requirements, their absence or contradictions; you can find:
1. INCOMPLETENESS -no defined actions for a specific combination of conditions
2. CONTRADICTION - defining in 2 different places of specification 2 different behaviours of the system
3. REDUNDANCY - defining same behaviour in 2 different places
STATE TRANSITION TESTING - APPLICATION
• technique used to check THE BEHAVIOUR OF THE COMPONENT OR THE SYSTEM
how it behaves over time and how it changes its state under the influence of various types of events
STATE TRANSITION DIAGRAM - model describing this behavioral aspect
DIFFERENT VARIANT OF STATE TRANSITION MODELS
1. FINITE AUTOMATON
2. FINITE STATE AUTOMATON
3. STATE MACHINE
4. LABELED TRANSITION SYSTEM
CONSTRUCTION OF THE STATE TRANSITION DIAGRAM
• graphical model as described in the UML standard
LABELED DIRECTED GRAPH
ELEMENTS:
1. STATES - represent possible situations in which system may be
2. TRANSITIONS - possible (correct) changes of states
3. EVENTS - phenomena, usually eternal to the system, the occurrence of which triggers the corresponding transitions
4. ACTIONS - actions that the system can take during the transition between states
5. GUARD CONDITIONS - logical conditions associated with transitions; a transition can be executed only if the associated guard condition is true
STATES IN STATE TRANSITION DIAGRAM
• STATE - an abstract, can denote a very high-level situation
e.g. system being in a certain application screen)
or low-level situations
e.g. system executing a certain program statement
• LEVEL OF ABSTRACTION OF STATE - depends on the model adopted, i.e. what the model actually describes and at what level of generality
• it’s assumed that when certain event occurs, the change of states is instantaneous
TRANSITION LABELS (arrows on the diagram)
• generally take form of:
event[guard condition]/ation
• if in a given case, the guard condition or action is not relevant or doesn’t exist, they can be omitted
TRANSITION LABELS - 3 FORMS
1. EVENT
2. EVENT/ACTION
3. EVENT[GUARD CONDITION]
- g.c. Allows transition to be executed if the condition holds
GUARD CONDITIONS
allow us to define 2 different transitions under the same event while avoiding non-determinism
EQUIVALENT FROMS OF STATE TRANSITION DIAGRAM
- STATE TABLE
individual rows and individual columns of the table represent successive states (respectively events, together with guard conditions if they exist)
in the cell at the intersection of the column and row corresponding to the specified state S and event E is written a target state - FULL TRANSITION STATE
table represent all possible combinations of states, events, and guard conditions
INVALID TRANSITIONS
• can be shown on state table and full transition table
• „missing arrows” in the start transition diagram represented in table by empty cells
invalid transition is any combination of states and event that doesn’t appear on the state transition diagram
TEST DESIGN: COVERAGE - 3 MOST POPULAR CRITERIA
- ALL STATES COVERAGE
• weakest coverage criterion
• the coverage items are the states
• all states coverage requires that the system has been in each state at least once - VALID TRANSITION COVERAGE (0-switch coverage or Chows coverage)
- most popular coverage criterion
- the coverage items are transitions between states
- valid transitions coverage requires that each transition defined in the state transtion diagram is executes at least once
- ALL TRANSITIONS COVERAGE
- the coverage items are all transitions indicated in the state table
- all valid transitions covered ans the execution of every invalid transition attempted. It’s good practice totes one such event per test (to avoid defect masking)
Other coverage criteria
• 1-SWITCH COVERAGE
every possible sequence of 2 consecutive valid transitions be tested
• N+1 consecutive transitions for any non-negative
• coverage of particular paths
• coverage of loops
COVERAGE IN STATE TRANSITION TESTING IN GENERAL
- the number of elements covered by the tests relative to the number of all coverage items defined by teh criterion
• usual requirement - DESIGN THE SMALLEST POSSIBLE NUMBER OF TEST CASES THAT ARE SUFFICIENT TO ACHIEVE FULL COVERAGE
CONVENTION TO DESCRIBE TEST CASES IN STATE TRANSITION TESTING
S1 (A) S2 (B) S3 (B) S4
S - STATES; A,B,C - EVENTS
„S (E) T” - T - TRANSITION
transition from state S under the influence of event E to state T
Steps in all stages coverage criteria:
TEST SCENARIO
Step|initial condition|event|expected result
1| S1 |A|transition to S2
2| S2 |B|transition toS1
3|S1 |C| to S3
4| S3 |B|to S4
5| S4 | |
TEST CASE VS TEST CONDTION
TEST CONDITIONS - individual states, but single TEST CASE can cover more than one test condition
TEST CASE
a sequence of transitions between states, starting with initial state and ending with final state (possibly interrupting this trek earlier if necessary)
EXAMPLE OF VALID TRANSITIONS COVERAGE CRITERION (S1-S4 states, and A,B,C events)
T1: S1 (A) S2
T2: S1 (C) S3
T3: S2 (A) S2
T4: S2 (B) S1
T5: S2 (C) S4
T6: S3 (B) S4 - THOSE ARE VALID TRANSITIONS
• design tests that would cover as many as possible of valid conditions
Test cases
TC1: S1 (A) S2 (A) S2 (B) S1 (C) S3 (B) S4
Transitions covered: T1, T3,T4,T2,T6
TC2: S1 (A) S2 (C) S4
COVERED: T1 (again) T5
And then cover INVALID TRANSITIONS
- if we managed to trigger an event not described in the model =>
a) failure - it should be impossible to trigger that
b) if we didnt change the state- correct behaviour
USE CASE TESTING
• requires a document that describes the interaction between so-called actors - most often: user and the system
• use case: description of steps and the system perform, which leads to the user obtaining some benefit
• each use case describe a SINGLE, WELL-DEFINED SCENARIO
• FOREACH STEP - ONE TEST CASE
USE CASE CONSTRUCTION
TECHNICAL INFO:
- pre-conditions
- exactly one, sequential main scenario
- (optional) markings of the locations of so-called alternative flows and exceptions
-post-conditions (describing the state of the system after successful execution of the scenario and the user benefit obtained)
• POSSIBILITY OF TESTING ALTERNATIVE FLOWS AND EXCEPTIONS
ALTERNATIVE FLOWS
causes an unexpected event to occur but allows the user to complete the use case, i.e. return to the main scenario
EXCEPTION
Interrupts the execution of the main scenario - not possible to complete the main scenario correctly; test case ends with error message and is aborted
USE CASE - NO BRANCHING, JUST LINEAR MAIN SCENARIO
DERIVING TEST CASES FROM A USE CASE
• minimum, to cover 100% od use cases
- one test case to exercise the main scenario, without any unexpected events
- enough test cases to exercise each exception and alternative flow
