Evaluation design I

Question 1

stages of evaluation

Answer 1

formative

process

outcome

The main purpose of evaluation design is to be as confidence as poss that any observed changes were cause by intervention, rather than by chance/other unknown factors

Question 2

formative eval

Answer 2

Before intervention

Acceptability and feasibility of intervention

Mainly qual, e.g. focus groups, in depth interviews

Question 3

formative eval research

Answer 3

Penn et al. (2018)

Question 4

Penn et al. (2018)

Answer 4

NHS diabetes prevention programme

eval - qual research

behav interventions

specification reflected evidence - framework for service provision

provides ev based behav intervention for prevention of T2D in high risk adults

Question 5

process eval

Answer 5

Measures how intervention delivered and received

Mixed qual and quan

Done along the way - make alterations if necessary

Question 6

process eval research

Answer 6

Sanchez et al. (2017)

Question 7

Sanchez et al. (2017)

Answer 7

improve understanding of underlying mechanisms that may impact results

prescribe healthy life intervention

moderate–>good performance on adoption, reach and implementation

Question 8

outcome eval

Answer 8

Measures whether intervention achieved objectives

Mainly quan

Question 9

outcome eval research

Answer 9

Ebert et al. (2018)

Question 10

Ebert et al. (2018)

Answer 10

internet and mobile stress management intervention and RCT

intervention v control

int = 7 sessions of problem solving and emotion regulation techniques

baseline v 6 months

cost-effective and lead to cost savings

Question 11

stages of evaluation research

Answer 11

Dehar et al. (1993)

Nutbeam (1998)

Question 12

Dehar et al. (1993)

Answer 12

formative - develop and improve programmes at an early stage

process - info on programme implementation, interpretation of outcomes and guiding future research

Question 13

Nutbeam (1998)

Answer 13

issues with definition and measurement of outcomes and use of eval methods

most powerful interventions = LT

technical problems from scientific rigour and advantages of less well defined content

combine and quan and qual

evals tailored to intervention

Question 14

cause and effect

Answer 14

Want to determine whether a cause-effect r’ship exists between intervention and outcome

Question 15

logic of causal inference

Answer 15

Under what conditions may we infer that a change in the DV (PA) was really caused by IV (intervention) and not by something else (envs etc.)

What are some of the most plausible rival explanations, and how do we rule them out?

Question 16

Logic of causal inference research

Answer 16

Rothman and Greenland (2004)

Hill (1975)

Question 17

Rothman and Greenland (2004)

Answer 17

causality often debated

more general conceptual model required

Question 18

Hill (1975)

Answer 18

https://journals.sagepub.com/doi/pdf/10.1177/003591576505800503)

Question 19

criteria for inferring causality

Answer 19

temporal r’ship

plausibility

strength of association

dose-response r’ship

reversibility

Question 20

temporal r’ship

Answer 20

Cause (intervention)must precede effect (increase in PA)

Question 21

plausibility

Answer 21

Association plausible, and more likely to be causal, if consistent with other knowledge

E.g. reasonable to expect people who receive exercise intervention will increase PA by greater amount than people without intervention

Question 22

strength of association

Answer 22

Strong association, as measured by effect size or relative risk, more likely to be causal than weak association

Question 23

dose-response r’ship

Answer 23

Occurs when changes in level of cause associated with changes in prevalence/incidence of effect

Greater adherence to exercise intervention (greater dose) the greater change in fitness

Incremental change in outcome, related to dose of intervention

1 intervention - adherence = linked to dose

Question 24

reversibility

Answer 24

When removal of cause (intervention) results in return to baseline in outcome (PA), likelihood of association being causal strengthened

Question 25

Internal validity

Answer 25

High internal val means that diffs observed between groups related to intervention tested in trial Means for example, that change in PA in study popn attributed to intervention and not to other factors, such as age, sex/social status IV has to be only thing influencing DV Extent to which you're able to say that no other variables, apart from the IV, caused a change in the DV Control and treatment group - treat them the same - see whether other variables influencing results

Question 26

Internal validity research

Answer 26

Halperin et al. (2015)

Question 27

Halperin et al. (2015)

Answer 27

internal val - degree of control exerted over confounds to reduce alternative explanations for results e.g. controlling ex protocols, prior training, nutritional intake, age, gender etc. less well controlled: instructions given, verbal encouragement, no. of observers and mental fatigue

Question 28

External validity

Answer 28

Described extent to which results of experiment of intervention can be generalised to target/general popn The extent to which the results of a study are generalisable to other situs/groups Good sampling needed - Representative of wider popn - Large enough to have adequate power - Exclusion criteria should relate to Q of interest

Question 29

External validity research

Answer 29

Leviton (2017)

Question 30

Leviton (2017)

Answer 30

need greater focus on external val goal is applicability methods - better description of int - combining of stat tools and logic to draw inferences about samples - sharper definition of theory - more systematic consultation of practitioners

Question 31

Validity research

Answer 31

Slack and Draugalis (2001) Blackman et al. (2013)

Question 32

Slack and Draugalis (2001)

Answer 32

establishing internal val of study based on logical process - provided by reports structure - methods describes how threats minimised - discussion assesses influence of bias threats: history, maturation, testing, instrumentation, regression, selection, exp mortality and interaction cog map used to guide when addressing threats - logical description of internal val problems threats source of extraneous variance external val addressed by delineating inclusion and exclusion criteria, describing subjects in terms relevant variables and assessing generalizability

Question 33

Blackman et al. (2013)

Answer 33

mobile health intervention see how well inform generalizability majority RCTs few studies addressed reach and adoption research designs need to focus on validity (internal and external)

Question 34

Types of evaluation design

Answer 34

Experimental Quasi-experimental Non-experimental

Question 35

Experimental

Answer 35

Compares intervention with non-intervention | Uses controls that are randomly assigned

Question 36

Experimental research

Answer 36

Sanson-Fisher et al. (2007) Bernal et al. (2018)

Question 37

Sanson-Fisher et al. (2007)

Answer 37

RCT often has limitations alternatives investigated

Question 38

Bernal et al. (2018)

Answer 38

control to minimise confounds consider confounds before deciding on control

Question 39

quasi-experimental research

Answer 39

Reeves et al. (2017) Handley et al. (2011)

Question 40

Reeves et al. (2017)

Answer 40

created checklist for quasi-exp designs - clustering of int as aspect of allocation/due to intrinsic nature of the delivery of intervention - for whom and when outcome data available - how intervention effect estimated - principle underlying control for confounding - how groups formed - features of study carried out after designed - variables measured before intervention

Question 41

Handley et al. (2011)

Answer 41

stepped-wedge and wait-list cross-over design relevant design features - creation of cohort over time that collects control data but allows all Ps to receive intervention - staggered intro of clusters - multiple data collection points - one-way cross over into intervention arm practical considerations - randomisation v stratification - training run in phases - extended time period for overall study completion

Question 42

non-experimental research

Answer 42

Thomson et al. (2001)

Question 43

Thomson et al. (2001)

Answer 43

systematic review of exp and non-exp housing intervention studies able to demo health gains even via non exp interventions some issues with generalizability

Question 44

experimental examples

Answer 44

RCT Pre-post design with randomised control group is one example of RCT

Question 45

experimental strengths

Answer 45

Can infer causality with highest degree of confidence

Question 46

experimental challenges

Answer 46

Most resource-intensive Requires ensuring min extraneous factors Challenging to generalise to "real world"

Question 47

quasi-experimental

Answer 47

Compares intervention with non-intervention | Uses controls or comparison groups that are not randomly assigned

Question 48

quasi-experimental examples

Answer 48

Pre-post design with a non-randomised comparison group

Question 49

quasi-experimental strengths

Answer 49

Can be used when you are unable to randomise a control group, but you will still be able to compare across groups and/ time points

Question 50

quasi-experimental challenges

Answer 50

Diffs between comparison groups may confound Group selection critical Moderate confidence in inferring causality

Question 51

non-experimental

Answer 51

Don’t use control/comparison groups

Question 52

non-experimental examples

Answer 52

Case control (post-intervention only): retrospectively compares data between intervention and non-intervention groups Pre-post with no control: data from one group compared before and after training intervention

Question 53

non-experimental strengths

Answer 53

Simple design, used when baseline data and/ comparison groups not available and for descriptive study May require least resources to conduct eval

Question 54

non-experimental challenges

Answer 54

Minimal ability to infer causality

Question 55

Evaluation in a nutshell (Bauman and Nutbeam, 2014)

Answer 55

see notes