Manual tasks: - (requirements analysis) - model creation - model validation - concretion implementation Automated tasks: - model veriﬁcation - test-case generation - test-case concretion - test-case execution - assignement of verdicts

Model Based Testing Flashcards by David Moling

Model-based testing

automatic generationof software test procedures

using models of system requirements and behavior

in combination with automated test execution

How well did you know this?

Not at all

Perfectly

Manual testing

+ easy & cheap to start
+ ﬂexible testing
- expensive every execution
- no auto regression testing
- ad-hoc coverage
- no coverage measurement

How well did you know this?

Not at all

Perfectly

Capture & Replay

+ auto regression testing
+ ﬂexible testing
- expensive ﬁrst execution
- fragile tests break easily
- ad-hoc coverage
- no coverage measurement

How well did you know this?

Not at all

Perfectly

testing with scripts

(e.g. JUnit)

+ auto regression testing
+ automatic execution
+/- test impl. = programming
- fragile tests break easily? (depends on abstraction)
- ad-hoc coverage
- no coverage measurement

How well did you know this?

Not at all

Perfectly

Test Scenarios

+ abstract tests
+ automatic execution
+ auto regression testing
+ robust tests
- ad-hoc coverage
- no coverage measurement
Examples: UML Testing Proﬁle (UTP) und Testing and Test Control
Notation (TTCN-3)

How well did you know this?

Not at all

Perfectly

Model based testing

explore a model that mirrors the SUT to generate test cases

How well did you know this?

Not at all

Perfectly

Model-Based Testing Pros/Cons

+ abstract tests
+ automatic execution
+ auto regression testing
+ auto design of tests
+ systematic coverage
+ measure coverage of model and requirements
- modelling eﬀorts

How well did you know this?

Not at all

Perfectly

MBT Workﬂow

*Manual tasks:**
(requirements analysis)
model creation
model validation
concretion implementation
*Automated tasks:**
model veriﬁcation
test-case generation
test-case concretion
test-case execution
assignement of verdicts

How well did you know this?

Not at all

Perfectly

Automata Learning Workflow

Automated tasks:

model creation
model veriﬁcation
test-case generation
test-case concretion
test-case execution
assignement of verdicts

Manual tasks:

model validation
implementation of test driver (mapper)

How well did you know this?

Not at all

Perfectly

reactive system

a software component which reacts to
the stimuli of its environment
Do not terminate
Test cases: sequences of events

event-based
controllable events
observable events

How well did you know this?

Not at all

Perfectly

mutation score

mutation score = #killed mutants / #mutants

How well did you know this?

Not at all

Perfectly

mutation testing goals

find test cases that maximize mutation score

How well did you know this?

Not at all

Perfectly

mutation testing problems

equivalent mutants
equivalence checking is hard

solutions
review of surviving mutants
generate mutants from model: model based mutation testing

How well did you know this?

Not at all

Perfectly

conformance

if an SUT alway passes a testcase, we have conformance

the test case conforms to the SUT

if we generate such a test case from a model → test case conforms to the model

if the SUT conforms to the model: test → model → SUT

Trace equivalence is the weakest notion of conformance

How well did you know this?

Not at all

Perfectly

mutation testing - non conformance

If the model does not conform to the SUT

and the Mutant conforms to the SUT

→ the model does not conform to the mutant

How well did you know this?

Not at all

Perfectly

mutation testing - Fault-Detecting

Mutation generated from the model

Kills the mutant if the test case conforms to the model and not to the mutant

it is a counterexample to conformance, hence the model does not conform to the mutant

How well did you know this?

Not at all

Perfectly

Transformational Systems

Model and Mutant interpreted as predicates describing state transformations (s → s’)

behaviour allowed by mutant but not by original model?

Explicit Conformance Checking

Build synchronous product modulo ioco between model and mutant

If mutant has additional
!output: ! fail sink state
?input: ! pass sink state

Extract test case covering fail state

Action Systems

Non-deterministic choice of actions

Actions are guarded commands that represent events
Loop over Actions
Terminates if all guards disabled

Actions are labelled

Action Systems - Semantics

labelled transition systems
predicative semantics

Predicative Semantics of Action Systems

The transition relation is translated to a constraint over state variables s and event-traces tr

Soundness

conformance, therefore all tests pass

No incorrect programs are deemed correct

Exhaustiveness

All tests pass → therefore conformance

A correct program is deemed correct ( ⇒ no spurious errors)

IOLTS

trace equivalent

A trace is an observable sequence of actions Two states are trace equivalent iﬀ they have the same traces Trace equivalence is the weakest notion of conformance

bisimilar

Two states are bisimilar iﬀ they simulate each other and go to states which are bisimilar Bisimulation is not suited for testing!

Suspension Automaton

adding loops for each quiescent state

IOCO

IUT ioco S iﬀ outputs (and quiescences) of the IUT are possible in S after an arbitrary suspension trace of S

RAISE

**R**igorous **A**pproach to **I**ndustrial **S**oftware **E**ngineering a method for software development a formal specification language RSL computer based tools

property oriented specification

no model often represented as **pure functions** and **stateless** programs PBT is a testing technique that tries to **falsify a given** property by generating **random input data** and checking the **expected behaviour** you define the properties that shold hold, like e.g. the length of two concatenated lists should be list1.length + list2.length

model based testing

we use a **model** of our subject under test (SUT) the model is _precise_ and tries to _mirror_ the SUT this makes most sense when the system has more non-functional complexity

timed automata

are automata with the notion of time (clocks) can be used to test real-time systems

black-box system

We can interact with it • Perform **inputs** • **Observe** actions • **Reset** it to initial state

active automata learning

automatically generate learned models (finite state machines) of black-box systems with the learned model we can perform model-based testing and verification

DNF

disjunctive normal form, can be used for automated theorem proving each variable appears exactly once in every conjunction

construction of Z

▪ X … input alphabet (collection of all stimuli) ▪ W … characterization set Z = 𝑊 ∪ 𝑋 ∙ 𝑊 ⋯ ∪ 𝑋^𝑚−𝑛∙ 𝑊

QuickCheck

property based testing random testing

Spec Explorer

Tool for testing reactive, object-oriented software systems State exploration produces a model automaton traversing the automaton → generated test ioco conformance relation then produces verdict

UPPAAL-TRON

ONLINE testing Testing Real-Time Embedded Software Generates and Executes Tests event-by-event in real time TRON: Replaces the environment of the IUT, simulates and monitors

ALERGIA

an identify any stochastic deterministic regular language determine the probabilities of the strings in the language configurable confidence interval

Model-based fuzz testing

black box testing technique the SUT is tested with an infinite number of randomized, invalid or unexpected inputs Model-based fuzz testing is used for randomizing sequential or concurrent behaviour of systems.