Lesson 11 (slides 1-21)

Question 1

What is the plan called that establishes data collection procedures and hypothosis testing?

Answer 1

Evaluation design AKA research design

Question 2

What provides structure for the major components of the study?

Answer 2

Evaluation design

• When, from whom, how, and what program procedures and measurements will be applied

Question 3

Overall, what is evaluation design attempting to address?

Answer 3

CAUSALITY

• Did the intervention/program cause the effects?

Question 4

What are the three design characteristics for improving causality?

Answer 4

1. Comparison/control group similar to the experimental/exposed group
2. Measurement of outcome variables before and after the intervention
3. Minimal threats to internal and external validity

Question 5

EVALUATION DESIGN NOTATION

R

Answer 5

Random assignment of an evaluation study participant (or unit) to a group

Question 6

EVALUATION DESIGN NOTATION

E

Answer 6

Experimental (intervention or treatment) group

Question 7

What does E1, E2, E3, …..En indicate?

Answer 7

Indicates planned exposure of the group to different intervention procedures.

Question 8

EVALUATION DESIGN NOTATION

C

Answer 8

Control (equivalent) group established only by random assignment

• Indicates NO exposure to an intervention or exposure to a minimum or standard intervention procedures

Question 9

EVALUATION DESIGN NOTATION

C

Answer 9

Comparison group established through any other method other than randomization

Question 10

EVALUATION DESIGN NOTATION

X

Answer 10

Intervention procedures applied to an E (experimental) group

Question 11

What does X1, X2, X3, …..Xn indicate?

Answer 11

Indicates an intervention consisting of multiple, different approaches

Question 12

EVALUATION DESIGN NOTATION

O

Answer 12

An observation or measurement to collect data.

Including:

• Tests
• Interviews
• Visual or audio ratings, or
• Record interviews

Question 13

What does O1, O2, O3,….. indicate?

Answer 13

Indicates multiple measurements at different times

Question 14

EVALUATION DESIGN NOTATION

T

Answer 14

Time when an observation, assignment to a group, or application of intervention procedures has occured.

Question 15

What does T1, T2,T3,…. Tn indicate?

Answer 15

Indicates repeated observations

Question 16

EVALUATION DESIGN NOTATION

n

Answer 16

Number of participants (sample)

- Could be in the E, C, or C group

Question 17

CATEGORIES OF EVALUATION DESIGN

Answer 17

Question 18

Is random assignment used?

• YES

Answer 18

Experimental design

Question 19

Is random assignment used?

• NO

Is there a comparison group or multiple observations?

YES?

NO?

Answer 19

YES

• Quasi-experiemental design

NO

• Non-experimental design

Question 20

CATEGORIES OF EVALUATION DESIGN

Includes one experimental group (E) but no

control group (C) or comparison group (C)

Answer 20

Non-experiemental design

Question 21

What are the 3 research designs?

Which one is the weakest and why?

Answer 21

The three research designs:

1. Non-experimental design
   
   • Weakest in regards to internal and external validity
2. Quasi-experiemental design
3. Experiemental design

Question 22

CATEGORIES OF EVALUATION DESIGN

• Includes an experiemental group (E) and a comparison group (C) created by methods other than random assignment
• Includes observations of both groups, both prior to and after application of intervention (X) procedures
  
  • Multiple waves of data collection
  • Does not account for all biases of internal validity

Answer 22

Quasi-Experimental Design

Question 23

CATEGORIES OF EVALUATION DESIGN

• Includes random assignment of experimental (E) group and a control (C) group
  
  • Observations of both groups prior to and after application of the intervention (X) procedures
  • Yields the most interpretable and defensible results of evidence of behavioral impact

Answer 23

Experimental Design

Question 24

What is the ‘Gold Standard’ in evaluation design and why?

Answer 24

Experimental design

• It yields the most interpretable and defensible evidence of behavioral impact
• Asserts the greatest degree of control over biasing factors that compromise internal validity of results

Question 25

What are the most commonly used evaluation (and research) designs?

Answer 25

1. One-shot case study 2. Retrospective Pretest 3. One-group pretest-posttest design 4. Time series design 5. Pretest-posttest control-group design 6. Case study design

Question 26

Which design is this? X--------O *(hint: How are paticipants faring after the services?)*

Answer 26

**One-shot case study** * (non-experimental)* - Survey of program participants after participation in the program

Question 27

What is the absolute most simplest form of outcome evaluation? ## Footnote WebRep currentVote noRating noWeight

Answer 27

**One-shot Design** Ex: How many people who attended a smoking cessation program are not smoking a month later? *- How participants are faring after the services.* WebRep currentVote noRating noWeight

Question 28

What are the primary weaknesses of one-shot design?

Answer 28

**Does not demonstrate change *_during_* or _as a result of_ the program** * No comparison * No multiple waves of data

Question 29

When is it best to use a one-shot design?

Answer 29

Best if used to answer descriptive research questions

Question 30

Why would it be difficult to establish observations are due to a disaster with a one-shot design?

Answer 30

Because there would be no comparison and no multiple waves of data collection

Question 31

Which design is this? X--------O \*\*\*respondents are asked to *_recall_* information before and after the intervention

Answer 31

**Retrospective Pretest** * Survey of program participants *_after_* participation in the program

Question 32

What are the primary weaknesses of the retrospective pretest?

Answer 32

**Recall Bias** - Relies on memory

Question 33

Which design is this? O1-------X--------O2

Answer 33

**One Group Pretest-Posttest Design** - Evaluator collects data *_before_* and *_after_* the program is conducted

Question 34

What are the primary weaknesses of the one group prestest-posttest design?

Answer 34

* Potential selection bias * Attrition (drop-out) * Potential measurment bias * Poor validity & reliability

Question 35

What needs to be considered with internal validity?

Answer 35

Is the intervention the only (or most) plausible explanation for the changes?

Question 36

What is internal validity?

Answer 36

Degree to which measured change can be attributed to the program

Question 37

What is external validity?

Answer 37

Extent to which the program can produce similar effects in other populations **- Is the program *generalizable*?**

Question 38

What are the two types of threats to internal validity?

Answer 38

Two primary sources of bias: 1. Non-program related sources of bias 2. Program related sources of bias

Question 39

What are the two sources of non-program related bias?

Answer 39

1. Maturation 2. Historical bias

Question 40

What are the three sources of program related bias?

Answer 40

1. Attrition 2. Selection bias 3. Measurement bias

Question 41

Which sources of bias are uncontrollable and which are somewhat controllable?

Answer 41

Uncontrollable: non-program related Controllable: program related