Knowledge clip 3 - Test phase

Question 1

What is a process model?

Answer 1

A process model represents an overview of the explanatory variables and the associations between the variables and the outcome variable

Question 2

Why do we need a process model?

Answer 2

A process model provides a structured, graphic overview of relevant variables and their relationships to each other and the outcome variable.

Question 3

How do we build a process model?

Answer 3

There is a set of heuristics and guidelines to use that guide us through the process of building a process model that fits our problem definition and analysis.

Question 4

What are the different types of associations?

Answer 4

• Direct
• Indirect
• Interaction
• Reinforcing
• Undermining

Question 5

Direct association

Answer 5

A causes B

A => B

Question 6

Indirect assocation

Answer 6

• A causes B through M
• Also called ‘mediation’
• Many behavioural psychological associations are in some shape or form indirect.

A => M => B

Question 7

Interaction association

Answer 7

• A causes B
• S influences the strength of association between A and B
• Also called ‘moderation’

A => B
S => (A => B)

Question 8

Reinforcing association

Answer 8

• A form of an interaction
• X strengthens the association between A and B
• The relationship between A and B is stronger when X has a higher value

A => B
X => (A => B)

Question 9

Undermining association

Answer 9

• A form of an interaction
• Y weakens the association between A and B
• The relationship between A and B is weaker when X has a higher value

Y => (A => B)

Question 10

What are the heuristics for building the process model?

Answer 10

• Make a list of the explanations/variables
  o This is the end point of your analysis phase
• Check that all the variables are psychological, specific, concrete, and continuous
• Check that all the variables are related to behaviour, attitudes, or affect
• Start by putting the outcome variable on the far right side of the model
• Work from right to left
  o Ask yourself: which variables affect the outcome variable?
  o On the left are the explanatory variables
• Draw arrows between the variables in your model to depict the direction of effects using + and –
• Make sure that the associations between your variables and outcome variables are not too remote
• Are the associations direct, indirect, or interactions?
  o You do this mostly based on the literature from the analysis phase
• Consider whether variables are also related to each other
• Keep it operational: +/- 10 variables (excl. outcome variable)
• Keep it operational: +/- 4 steps between outcome and far left of the model
  o You don’t take more than 4 steps, because the further away your variable is from the outcome variable, the more diluted, and thus weaker, the association becomes. This makes it less interesting to focus on.

Question 11

The heuristics of creativity in employees.

A list of factors that affect the creativity in employees based on the analysis phase:

• Stress
• Mood
• Cognitive resources
• Time pressure
• Technical skills
• Number of tasks
• Competition
• Self-efficacy
• Etc.

Answer 11

• You start by putting your outcome variable on the far right
  o In this case: creativity score
• Then you put the variables that are most directly related to your outcome variable, together with their direction (+/-)
  o In this case: cognitive resources, mood and mental fatigue
• Then you start working further to the left
  o In this case: stress
  o Further to the left in this case: competition, time pressure, work pressure, self-efficacy on stress
• If a variable has an indirect effect on the outcome variable, this doesn’t mean that it can’t also have a direct effect, so you should look at this.
• Finaly you look at possible moderators

Question 12

Why do you have to select variables

Answer 12

The list of variables from the analysis is longer than the list of variables that you want to include in the model, in order to keep it operational and workable.

Question 13

How do you select variables?

Answer 13

Psychological

• Is it a psychological variable?
• If it’s not, it automatically becomes less interesting for a psychological process model.

Modifiability

• Can the variable be changed/influenced?
• If a variable is very static it might not be that interesting because you’re looking for variables to target with your intervention.

Strength of association

• Is the association strong enough to be interesting?
• It should not be too weak or too remote.

Context

• Is the context of evidence transferable?
• Is the context of the evidence that you found still applicable to your case?

Evidence base

• Is there enough high level evidence for the association to be supported in your model?

Question 14

What does testing the model mean?

Answer 14

Testing the model means that you zoom in on all the different arrows and you need to be able to support all these arrows by the evidence that you have.

Question 15

The evidence that you have can have different levels of evidence base. What are these levels of evidence?

Answer 15

Opinions:

• The sixth level; the lowest level
• It might be from experts but they’re only opinions

Case studies:

• The fifth level
• Some level of evidence but not so much, especially if you only have one or two

Observational studies, pre-post measurements:

• The fourth level
• Some level of evidence but not so much, especially if you only have one or two

Non-randomized controlled trial

• The third level

Randomized controlled trials

• The second level

Systematic reviews and meta-analysis

• The first level; the highest level
• They support the association that you have represented in your model
• These are always good to look at because they tend to summarize the evidence that is out there.
• They give you information about the strength of the association
• They often give information about the evidence base that underlies the review or meta-analysis and how strong and what level that is

Question 16

P-values vs effect sizes

Answer 16

We want to look at the actual statistical effect sizes or the strength of association that underlies each of the arrows in our model.

Go beyond the p-value and look at the effect sizes

• The p-value tells us something, but not so much about the strength of association.

Statistical significance testing cannot evaluate result importance. Cohen (2994) observed that the statistical significance test ‘does not tell us what we want to know, and we so much want to know what we want to know that, out of desperation, we nevertheless believe that it does!’

So, p-value tells us something about the statistical significance, which is important, but we are more interested in the strength of association.

Question 17

Effect size

Answer 17

A statistic that indicates to what extent sample results differ from the expectations as phrased in the null hypothesis (i.e., no difference between groups, no association between variables).

I does not tell us if it’s statistical significance but how strong the relation is. They are often related to each other.

Effect size (ES) can be in original measurement units (e.g., ms, scores on a scale) or standardized (e.g., Cohen’s d, beta, eta, r).

Question 18

Is the association/effect size based on a large quantity of studies (e.g., a meta-analysis) or a single or small set of studies?

Answer 18

• One effect size is no effect size
• The more you have to support the effect size that you are reporting, the stronger your evidence base is

Question 19

What do you need to know about the number of people when looking at effect sizes?

Answer 19

How large is N?

• How many people are included in the different studies?

Who is N?

• Who are the participants?
• This tells you something about how generalizable the effects are

Question 20

Tips and tricks of the process model

Answer 20

Find psychological mediators and moderators.

Are all variables concrete?

• Not ‘attitudes’ but ‘attitudes towards stair use’
• Not ‘demands’, but ‘experienced time pressure’

Stick to one perspective in your model

• E.g.., when researching ‘bullying’ focus on the bully, the victim, or the bystanders, but not all of them.

Start with the problem: what needs to change?

Support all associations in the model.

Be complete, but keep it workable.

Question 21

What is the end product of the test phase?

Answer 21

A model that represents the different associations between variables from the analysis phase and the outcome variable.