Model Based Testing and Runtime Verification

Question 1

The behaviors allowed by the implementations

Answer 1

The behaviors allowed by the implementations are a subset of those allowed by the specification.

Question 2

The Testing Perspective

Answer 2

Test -> Implementation -> Oracle

Question 3

The Runtime Verification Perspective

Answer 3

User -> Implementation -> Monitor

Question 4

Monitors

Answer 4

require generalization, making things harder. Hence we need a specification language (other than the programming language itself).

Question 5

A Specification Language - Finite State Automata

Answer 5

Make a list of the inputs
For each input, note the situations in which the input can be applied and when it can’t
For each situation, note further situations when the input causes different behaviors
What variables do we need to model all possible situations

Question 6

Runtime Verification

Answer 6

Basically monitoring the system under test through an RV tool.

Question 7

RV architecture

Answer 7

Monitoring guarantees that the system never advances beyond failure.

Upon failure one can attempt to fix the problem since we are guaranteed to stop immediately after the failure.

The cost in overheads can be prohibitive if the volume of monitored events is large.

The system can break real-time properties because of monitor-caused delays.

Question 8

Building the Architecture

Answer 8

Separate specification from system requirements, this requires additional work to weave the two together.

Question 9

Separation of Concerns: Aspect-Oriented Programming

Answer 9

Modular abstraction techniques (OO design aims at abstracting models, sharing common parts across the implementation).

Sometimes, one modularization strategy forgoes another, requiring code replication and the merging of business logic and support code.

Aspect-Oriented programming gives ways of changing code across a whole system in a modular way.

Consider adding a logging feature for all types of money transfers, which can be toggled on or off; by adding a line to each transfer method or by using an AOP tool.

Question 10

Property Specification Languages: Automata

Answer 10

Finite State Automata can be a good way of allowing the specification of consequentiality behavior. Access to system data enhances expressiveness and interaction with the system.
Parameterization allows us to have a separate monitor per use.

Question 11

Determinism

Answer 11

Monitoring automata should be deterministic. Transitions and automata are ordered.

Question 12

RV Tool

Answer 12

Larva - used in JAVA using Symbolic Timed Automata.

Question 13

Property Specifications

Answer 13

DATEs (Dynamic Automata with Timers and events): Automata based. Dynamic creation of new properties (automata). Replication of properties per object instance. Triggering of transitions based on program method invocations, program exceptions, real time events, and channel communication between automata.

Question 14

Model Based Testing

Answer 14

Testing is mainly done manually which is tedious and error prone, automation is better.
Bugs occur at the model and implementation. We can check correspondence to find bugs.

Question 15

Generic Model Based Testing

Answer 15

Test suite generation: Use the model to generate meaningful inputs/sequences of inputs (Decide the strategy of traversing the model).
Prioritizing Test Cases: Random, Greedy Algorithm, Lookahead.
Executing Tests: Usually automated but might need some scaffolding/adapter.

Question 16

Test Suite Execution

Answer 16

Offline: Generate once, execute many times (ex. Regression Testing)
Online: Generate on-the-fly, Ideal for time-bound testing (ex. Test for a whole night). Testing systems whose environment cannot be easily simulated (such as 3rd party systems).

Question 17

Oracle Problem

Answer 17

Models necessitate automation of oracle since model based testing is usually very fast and comprehensive (in terms of system functionality). Ex. Assertions can be added to FSMs to check the system still corresponds with the model.

Question 18

Measuring Test Effectiveness

Answer 18

Structural Code Coverage - Aims to exercise code or model concerning some coverage goal.
Coverage Measures:
States visited
Transitions visited
Transition pair combinations

Question 19

Test Failures

Answer 19

Fix the models because it isn’t correct
Fix the system because it isn’t correct

Question 20

JUNIT Model

Answer 20

Define a model
Normally using an FSM with variables, conditions and actions
Build graph of possible transitions
Generates abstract teszta.
- Add operation on SUT
- Update any tracked variables
- Add JUNIT assertions on SUT after applying actions

Question 21

Guards

Answer 21

You can add a guard to indicate when it is possible to choose a particular action. Add a method that returns a Boolean. It should have the same name as the action method name and end with the word Guard.

Question 22

Generating Test Cases

Answer 22

• Random
• Greedy Tester: Decides at the current state which transitions are yet to be visited
• Lookahead tester: Looks ahead to find unvisited transitions