Model Based Testing

Question 1

Model-based testing

Answer 1

automatic generationof software test procedures

using models of system requirements and behavior

in combination with automated test execution

Question 2

Manual testing

Answer 2

+ easy & cheap to start
+ flexible testing
- expensive every execution
- no auto regression testing
- ad-hoc coverage
- no coverage measurement

Question 3

Capture & Replay

Answer 3

+ auto regression testing
+ flexible testing
- expensive first execution
- fragile tests break easily
- ad-hoc coverage
- no coverage measurement

Question 4

testing with scripts

Answer 4

(e.g. JUnit)

+ auto regression testing
+ automatic execution
+/- test impl. = programming
- fragile tests break easily? (depends on abstraction)
- ad-hoc coverage
- no coverage measurement

Question 5

Test Scenarios

Answer 5

+ abstract tests
+ automatic execution
+ auto regression testing
+ robust tests
- ad-hoc coverage
- no coverage measurement
Examples: UML Testing Profile (UTP) und Testing and Test Control
Notation (TTCN-3)

Question 6

Model based testing

Answer 6

explore a model that mirrors the SUT to generate test cases

Question 7

Model-Based Testing Pros/Cons

Answer 7

+ abstract tests
+ automatic execution
+ auto regression testing
+ auto design of tests
+ systematic coverage
+ measure coverage of model and requirements
- modelling efforts

Question 8

MBT Workflow

Answer 8

• *Manual tasks:**
• (requirements analysis)
• model creation
• model validation
• concretion implementation
• *Automated tasks:**
• model verification
• test-case generation
• test-case concretion
• test-case execution
• assignement of verdicts

Question 9

Automata Learning Workflow

Answer 9

Automated tasks:

• model creation
• model verification
• test-case generation
• test-case concretion
• test-case execution
• assignement of verdicts

Manual tasks:

• model validation
• implementation of test driver (mapper)

Question 10

reactive system

Answer 10

a software component which reacts to
the stimuli of its environment
Do not terminate
Test cases: sequences of events

event-based
controllable events
observable events

Question 11

mutation score

Answer 11

mutation score = #killed mutants / #mutants

Question 12

mutation testing goals

Answer 12

find test cases that maximize mutation score

Question 13

mutation testing problems

Answer 13

equivalent mutants
equivalence checking is hard

solutions
review of surviving mutants
generate mutants from model: model based mutation testing

Question 14

conformance

Answer 14

if an SUT alway passes a testcase, we have conformance

the test case conforms to the SUT

if we generate such a test case from a model → test case conforms to the model

if the SUT conforms to the model: test → model → SUT

Trace equivalence is the weakest notion of conformance

Question 15

mutation testing - non conformance

Answer 15

If the model does not conform to the SUT

and the Mutant conforms to the SUT

→ the model does not conform to the mutant

Question 16

mutation testing - Fault-Detecting

Answer 16

Mutation generated from the model

Kills the mutant if the test case conforms to the model and not to the mutant

it is a counterexample to conformance, hence the model does not conform to the mutant

Question 17

Transformational Systems

Answer 17

Model and Mutant interpreted as predicates describing state transformations (s → s’)

behaviour allowed by mutant but not by original model?

Question 18

Explicit Conformance Checking

Answer 18

Build synchronous product modulo ioco between model and mutant

If mutant has additional
!output: ! fail sink state
?input: ! pass sink state

Extract test case covering fail state

Question 19

Action Systems

Answer 19

Non-deterministic choice of actions

Actions are guarded commands that represent events
Loop over Actions
Terminates if all guards disabled

Actions are labelled

Question 20

Action Systems - Semantics

Answer 20

1. labelled transition systems
2. predicative semantics

Question 21

Predicative Semantics of Action Systems

Answer 21

The transition relation is translated to a constraint over state variables s and event-traces tr

Question 22

Soundness

Answer 22

conformance, therefore all tests pass

No incorrect programs are deemed correct

Question 23

Exhaustiveness

Answer 23

All tests pass → therefore conformance

A correct program is deemed correct ( ⇒ no spurious errors)

Question 24

IOLTS

Answer 24

Question 25

trace equivalent

Answer 25

A trace is an observable sequence of actions

Two states are trace equivalent iff they have the same traces

Trace equivalence is the weakest notion of conformance

Question 26

bisimilar

Answer 26

Two states are bisimilar iff they simulate each other and go to
states which are bisimilar

Bisimulation is not suited for testing!

Question 27

Suspension Automaton

Answer 27

adding loops for each quiescent state

Question 28

IOCO

Answer 28

IUT ioco S iff outputs (and quiescences) of the IUT are possible in S after an arbitrary suspension trace of S

Question 29

RAISE

Answer 29

Rigorous Approach to Industrial Software Engineering

a method for software development
a formal specification language RSL
computer based tools

Question 30

property oriented specification

Answer 30

no model

often represented as pure functions and stateless programs

PBT is a testing technique that tries to falsify a given property by generating random input data and checking the expected behaviour

you define the properties that shold hold, like e.g. the length of two concatenated lists should be list1.length + list2.length

Question 31

model based testing

Answer 31

we use a model of our subject under test (SUT)

the model is precise and tries to mirror the SUT
this makes most sense when the system has more non-functional complexity

Question 32

timed automata

Answer 32

are automata with the notion of time (clocks)
can be used to test real-time systems

Question 33

black-box system

Answer 33

We can interact with it
• Perform inputs
• Observe actions
• Reset it to initial state

Question 34

active automata learning

Answer 34

automatically generate learned models (finite state machines) of black-box systems

with the learned model we can perform model-based testing and verification

Question 35

DNF

Answer 35

disjunctive normal form, can be used for automated theorem proving
each variable appears exactly once in every conjunction

Question 36

construction of Z

Answer 36

▪ X … input alphabet (collection of all stimuli)
▪ W … characterization set

Z = 𝑊 ∪ 𝑋 ∙ 𝑊 ⋯ ∪ 𝑋^𝑚−𝑛∙ 𝑊

Question 37

QuickCheck

Answer 37

property based testing

random testing

Question 38

Spec Explorer

Answer 38

Tool for testing reactive, object-oriented software systems
State exploration produces a model automaton
traversing the automaton → generated test
ioco conformance relation then produces verdict

Question 39

UPPAAL-TRON

Answer 39

ONLINE testing
Testing Real-Time Embedded Software
Generates and Executes Tests event-by-event in real time

TRON: Replaces the environment of the IUT, simulates and monitors

Question 40

ALERGIA

Answer 40

an identify any stochastic deterministic regular language
determine the probabilities of the strings in the language

configurable confidence interval

Question 41

Model-based fuzz testing

Answer 41

black box testing technique

the SUT is tested with an infinite number of randomized, invalid or unexpected inputs
Model-based fuzz testing is used for randomizing sequential or concurrent behaviour of systems.