Designing training experiments

Question 1

THE AIM

Answer 1

In Experimental studies the goal is to make an
inference about the causal effect of the IV (e.g., level of physical activity) on the DV (e.g., health)

This causal link can be clouded by certain factors:
confounders and mediators

Question 2

CONFOUNDERS

Answer 2

An alternative IV that can explain the effect you

observed

Question 3

MEDIATORS

Answer 3

An IV that partially determines

the strength of the relationship between your IV and DV

Question 4

How to deal with confounders and mediators

Answer 4

Hold constant in sample
 Choose participants so your groups are matched for
potential confounders (e.g., age, gender, etc).
 Hold constant in analysis
 Include potential confounding variables in your    analysis as covariates

Question 5

classic experimental designs

Answer 5

pre-post single

pre-post multiple

Question 6

The 5 design issues

Answer 6

Pre/post test
Training
Control group
Transfer 
Retention

Question 7

The Hawthorne Effect

Answer 7

A phenomenon where behaviour is altered due to
the knowledge that one is being observed or monitored

It is a motivational effect, and therefore often leads
to improvements in performance (temporary)

Question 8

Hawthorne effect

Wolfe and Michaud (2010)

Answer 8

264 rheumatoid arthritis patients
received comprehensive care in a clinical trial, including free treatment, and received ordinary RA care by their non-study physicians after the trial

Measured four variables pre-trial, post-trial, and eight months after post-trial (retention)

Almost half of the improvement noted in the clinical
trial HAQ score disappeared on entry to a non-sponsored follow-up study (8 months after post-trial), and from 23% to 44% of improvements in pain, patient global, and fatigue also disappeared

Question 9

The Placebo Effect

Answer 9

The belief that you are receiving something to help performance can often increase motivation, confidence, etc.

Particularly problematic if your control group receives no training

Question 10

The Placebo effect

Guillot et al (2012)

Answer 10

Examined the effect of motor imagery (MI) on tennis
serve performance

3 training groups:
 Control – physical practice only
 MI – physical practice plus motor imagery
 Placebo – same as MI except were also told that a
custom-made racquet was developed for them

Both MI and placebo had significantly higher % of
successful serves than the control group

Only the placebo group had significantly more
accurate serves than the control group

Question 11

Regression to the mean

Answer 11

Training programs often focus on players who are performing below their normal ability

There is a natural tendency for a player to move towards his/her average (mean) performance regardless of training method

Question 12

Regression to the mean

Does sacking the manager improve football team’s performance? Anderson and Sally (2013)

Answer 12

“Even without sacking the manager, the performance of the control group bounces back in the same fashion and at least as strongly as the performance of the clubs that fired their managers (experimental group)”

Question 13

The replication problem

Answer 13

findings in psychology have been called into question when follow-up studies failed to replicate them

May be due to:
 False positive in original
 False negative in replication
 Correct but different results in both due to unintended methodological differences

Question 14

The replication problem study by open science collaboration (2015)

Answer 14

100 experimental and correlational studies published in psychology were replicated

36% of replicated studies were significant, compared to 97% in original studies

Mean effect size was less than half those of the original results

Question 15

What are pre-mortems?

Answer 15

Pre-mortems occur before a project commences
and allow individuals to identify what might cause the project to “die”

Shown to be effective in combating overconfidence
in project planning above other techniques (e.g. ‘Pros/Cons Generation’)

Question 16

Five stages of pre-mortems?

Answer 16

Step 1: The research team is explained the plans and design of the project

Step 2: The team leader announces that the project failed spectacularly

Step 3: Each individual independently lists every possible reason they can think of for the failure

Step 4: The leader asks each team member to read one (different) reason from their list until all ideas have been recorded

Step 5: The new list can now be reviewed and strategies developed to address these ideas

Question 17

Why are pre-mortems useful?

Answer 17

Allows for the identification of potential problems early on

Helps reduce reckless attitudes that can be evident in individuals who are overinvested in a project

Allow team members to feel valued for their ideas, and gives others the chance to learn from them

Sensitizes the team to pick up early signs of trouble once the project begins