Evidence-based evaluation

Question 1

What kind of evaluations are there to assess the strategy’s effectiveness?

Answer 1

1. EMMIE Framwork
2. The Maryland Scale (SMS)

Question 2

EMMIE rates each intervention against 5 dimensions. What are these dimensions? And what do they do?

Answer 2

1. Effect –> what is the impact on the crime.
2. Mechanisms/mediation –> how does it work; What is it about the intervention that could explain its effect.
3. Moderators –> where does it work; what are the circumstances and contexts where the intervention is likely to work or not work.
4. Implementation –> how to do it; the conditions that should be considered when implementing an intervention.
5. Economic-costs –> how much does it cost; direct and indirect.

Question 3

What are the 4 criteria of methodological quality? And what does it mean?

Answer 3

1. Statistical conclusion validity –> the cause precedes the effect.
2. Internal validity –> the cause is related to the effect.
3. Construct validity –> there is no other explanation for the effect.
4. External validity –> the results can be applied to different people/places/times.

Question 4

What are the main threats to the statistical conclusion validity?

Answer 4

• Low statistical power (e.g., due to sample size)
• Using inappropriate statistical methods

Question 5

What are threats to the internal validity?

Answer 5

• History: the effect is caused by some event occurring at the same time as the intervention.
• Maturation: the effect reflects a continuation of preexisting trends (normal human development).
• Instrumentation: the effect is caused by a change in the method of measuring the outcome.
• Testing: the pretest measurement causes a change in the posttest measure.
• Differential attrition: the effect is caused by differential loss of units from experimental compared to control conditions.

Question 6

What are the main threats to construct validity?

Answer 6

• Whether the intervention successfully changed what it was supposed to change (e.g., implementation issues, treatment fidelity)
• The accuracy and reliability of the outcome measure (e.g., whether police-recorded crime rates reflect actual crime rates).

Question 7

Bad science = bad policy
What can we do?

Answer 7

• Adhere to scientific standards.
• Collect and evaluate studies
• Examine theoretical mechanisms
• Test, retest, repeat

Question 8

The Maryland Scale (SMS) ranks research design from 1 to 5, what are these levels? And which are (in)appropriate?

Answer 8

Level 1: correlation, single point in time –> inappropriate
Level 2: before and after, no control group –> inappropriate
Level 3: before and after, with control group –> appropriate
Level 4: before and after, multiple control groups, and control for other variables –> appropriate
Level 5: randomized assignment to comparable groups; randomized controlled trials (RCT) –> appropriate

Question 9

Which questions do you have to ask, after ranking the SMS to an intervention/evaluation?

Answer 9

Statistical conclusion validity:
1. Was the statistical analysis appropriate?
2. Did the study have low statistical power to detect effects because of small samples?
3. Was there a low response rate or differential attrition?
Construct validity:
4. What was the reliability and validity of measurement of the outcome?

Question 10

What do you have to do if you have serious implementation problems? (SMS)

Answer 10

You have to deduct the level of the Maryland Scale by one point, for example when the attrition is really high.

Question 11

Reviewing the collective evidence of the SMS, what works? What doesn’t work? What is promising?

Answer 11

What works –> at least two level 3-5 evaluations showing effects.
What doesn’t work –> at least two level 3-5 evaluations showing no effects.
What is promising –> at least one level 3-5 evaluation showing effects.

Question 12

What are the main threats to external validity?

Answer 12

Differences in how causal relationships vary across types of people, settings, interventions, or outcomes.