Knowledge clip 5 - Success phase

Question 1

The succes phase

Answer 1

The final phase of the PATHS.

Evaluating the intervention in terms of process and effect.

You want to have a plan on how to evaluate your intervention

• You need to think about this beforehand
  o E.g., while phrasing your outcome variable

Question 2

What are the types of evaluations that you can focus on?

Answer 2

1. Effectiveness
2. Process

Question 3

What do you look at with an effect evaluation?

Answer 3

Outcome variable

• How much did your outcome variable change over time after implementing the intervention?
• This is the primary effect
  o You have an outcome variable that you want to change and if this changes it will at least partially resolve the problem that you started with.

Target variable

• How much did your target variable (selected from the balance table in the Help phase) change?
• This helps you to draw conclusions about whether your intervention had the impact that you expected it to have.

Measurement

• A baseline measurement or control group strengthens your effect evaluation.
• If you only have a pre- and post-measurement, this is less strong than if you have a control group.
• Having a baseline is the bare minimum to be able to assess any changes in the outcome variable.

Question 4

The effect ladder

Answer 4

Part of the effect evaluation.

If you look at the effectiveness and the evidence base that you need to be able to draw conclusions, you can look at the effect ladder.

You look at the strength of evidence, the type of research needed and the qualification of the intervention.

1 = low and 5 = high

Question 5

Level 1 in the effect ladder

Answer 5

Strength of evidence = descriptive clues, no evidence for effects

Type of research needed = descriptive research, interviews, Delphi-panels

Qualification of intervention = conditional

Question 6

Level 2 in the effect ladder

Answer 6

Strength of evidence = theoretical clues for evidence

Type of research needed = literature research

Qualification of intervention = promising

Question 7

Level 3 in the effect ladder

Answer 7

Strength of evidence = first empirical clues for evidence

Type of research needed = pre-test post-test design or post-test only design

Qualification of intervention = achieving goals

Question 8

Level 4 in the effect ladder

Answer 8

Strength of evidence = plausible empirical clues for evidence

Type of research needed = comparisons with a benchmark

Qualification of intervention = plausible

Question 9

Level 5 in the effect ladder

Answer 9

Strength of evidence = strong empirical clues for evidence

Type of research needed = experimental studies or quasi-eperimental studies

Qualification of intervention = effective

Question 10

Feasibility in the effect ladder

Answer 10

It’s not always possible to have experimental studies.

It’s not always ethical to have experimental studies.

So sometimes you are ‘stuck’ at a lower step on the ladder.

Question 11

Process evaluation

Answer 11

You try to move away from the effects themselves and look at the things that have happened surrounding your intervention.

Question 12

What are the examples of process evaluation?

Answer 12

• Completeness of implementation
• Exposure to the intervention
• Satisfaction with the intervention
• Recruitment
• Context

Question 13

Completeness of implementation (process evaluation)

Answer 13

Were all the elements of the intervention actually used and implemented?

Sometimes you are dependent on different stakeholders/parties that all have to implement elements of your intervention.

Question 14

Exposure to the intervention (process evaluation)

Answer 14

To which extent did members of the target group engage with the intervention?

Question 15

Satisfaction with the intervention (process evaluation)

Answer 15

To which extent did members of the target group positively evaluate the intervention?

If they did not like it, this can play a part in how effective the intervention has been in changing behaviour.

Question 16

Recruitment (process evaluation)

Answer 16

What procedures were used to attract members of the target group to the intervention?

Question 17

Context (process evaluation)

Answer 17

Which environmental factors influenced the implementation of the intervention or the outcomes?

Were there any situations or instances in which the context might have been more influential than you would have expected in the beginning?

Question 18

Evaluation plan

Answer 18

Timeline

• Plan your evaluation before implementing the intervention

Outcome variable

• It may be necessary to measure your outcome variable before implementing the intervention (pre-test/baseline)

Feasibility

• Think of the resources you may need to reach your target group, conduct measurements, etc.

So, the evaluation plan is a little intervention plan by itself.