5. Understand research designs Flashcards
PRE-EXPERIMENTAL DESIGNS:
One shot-case Study
Diagram of the design: * M
Examples: Observe that somebody jogs (treatment) and notice that person is slim. Assume jogging caused slimness (but could have been skinny before jogging)
Threats to inference: can’t observe a change, because there’s no premeasurement can’t attribute observed result to the treatment with any confidence “Such studies have such a total absence of control as to be of almost no scientific value.” Campbell & Stanley
Natural experiment without premeasurement: Measure the DV, then sort data based on variation on the IV
Test group * M
Control group M
A design in which no measurement are made before subjects are exposed to the IV
1.Measure the DV for subjects of whom have been exposed to the DV (test group) and some of whom have not been (control group)
2.If the DV for the different groups are different, ascribe this to the effect of the IV
Examples:
Tax reform mail -
1. Observe how Senators voted on tax reform. 2. See what kind of mail senators got, who voted for and against. 3. Attribute voting decision to kind of mail received
Threats to inference:
• the membership of the test and control groups may not be equivalent at beginning because no premeasurement (selection)
• the two groups may differ in their propensity to change in response to the treatment (selection)
i.e. the groups could be different in ways not revealed by premeasurement.
E.g. The two groups of Senators may differ in their receptiveness to the mail
🡪 Any observed differences between groups may be due to selection effects, not the IV or treatment
• also confounding factors - can’t rule out other factors occurring at same time, affecting groups unequally i.e. lobbying
Observation (no control group) also known as one group pre-test/ post-test:
Diagram: M * M
Example: From 1980-88 President Reagan increased US defense spending. Between 1989 -1991, the USSR collapsed. So Reagan caused the collapse of the USSR. NOT SO FAST…
Threats to inference:
other events in the same time period may have caused the effect (history)
With this design, potentially anything that happened or changed between the two measurements could explain the second measurement M.
Can’t rule out other factors as causes of the observed change in DV
Example: Eating garlic to cure a cold .
Threats to inference: - maturation (effect of passage of time) A particular problem with studies of children
Regression to the mean when evaluating policy:
Diagram: M * M
The problem is called regression to the mean. lf we assume that essentially everything we observe has some element of randomness to it-that is, in addition to its true core value it varies somewhat fron1 time to time-regression to the mean will always be present. Consider a student in a course, for example. When she is tested, her measured level of knowledge is generally about right, but if she has had a bad day (isn’t feeling good, has bad luck in the instructor’s choice of questions, etc.), she will score somewhat below her usual level; on a good day, she will score somewhat higher. Note that this sort of random variation in a measure is closely related to what we referred to as unreliability in Chapter 4.
An antidote for regression to the mean, interrupted time series: If the measurement that came just before the intervention was unusually high, the measurements preceding it should tend to be lower than it is. The test for whether the intervention has had an effect in this design is whether the average of the several measurements preceding the intervention differs from the average of the measurements following the intervention.
QUASI-EXPRIMENTAL DESIGNS
Natural Experiment:
A natural experiment is an empirical study in which individuals (or clusters of individuals) are exposed to the experimental and control conditions that are determined by nature or by other factors outside the control of the investigators.
Diagram: Test group M * M
Control group: M M
Example: Organizing the poor
1. The study’s authors survey poor people to measure their interest in politics (M)
2. Organizers conduct a campaign (*) to organize poor. Some of those surveyed are not contacted, but people are not randomly assigned to be contacted. The campaigners picked them, not study’s authors.
3. Study authors survey same people as before (M). Those contacted by campaigners more interested than those NOT contacted.
Threats to inference:
The test and control groups may not be equivalent, even if same scores on pre-measurement. They may be different on something unobserved. (selection effects)
For example, the test and control groups may differ on variables we haven’t measured, such as motivation, leading to a greater predisposition to change.
(Note: haphazard selection of the test and control groups is not RANDOM assignment to treatment and control)
The use of a control group in this design fixes the threats to inference from
history, measurement, regression to the mean and maturation
Time series:
MMM * MMM
Example: Change in legislation such as introducing a helmet law. Measure death rates before and after change in the law
Threats to inference:
• Other events in the same time period may have caused (history)
• (Multiple measurements fix regression to mean)
Control group time series:
Test group: MMM * MMM
Control group: MMM MMM
Example: Change in legislation in one jurisdiction, but not in the other
Threats to inference:
• Differences between two groups may be due to selection, not the treatment. The two jurisdictions may not be comparable.
• There may be confounding factors that differ between the two jurisdictions
• (Design controls for history (events affecting both jurisdictions the same way) and regression to the mean)
TRUE EXPERIMENTAL DESIGNS
True Experiment
Test group R M * M
Control R M M
Treatment* must be randomly assigned, which determines who is in test and control groups
Example: Presidential lobbying
1. conduct straw poll of senators (M)
2. randomly assign some to be lobbied (R)
3. lobby that group (*)
4. measure final votes (M) and compare two groups
Threats to inference:
• Controls for most threats to inference (internal validity).
• controls for differences between the two groups that we don’t know about or can’t measure (selection bias)
• If sample is large enough and randomly assign treatment, only difference between the groups will be the treatment variable.
• Very likely that difference between treatment and control groups is the treatment (IV) and not something else.
• True experiment is the only research design that let’s us conclude “The IV caused the DV.”
