Lec. 20 - Critical Thinking in Quasi Experimentation (article)

Question 1

General note

Answer 1

On Canvas it says that we have to read carefully the article by Shadish, and use the other two articles as references, but not study in these last two articles information that was not also in the lecture.
Therefore, these flashcards are on the Shadish article, while if there is important information from the other two articles it is together with the flashcards from the lecture.
Peace and love!

Question 2

Project Head Start example

Answer 2

• they created a program for disadvantaged children called “Head Start”
• they tested academic achievements of the eligible children and compared them to children in same grade, same gender and that also applied to Head Start but were not eligible
  → this is a quasi experiment (not ethical to randomly assign children to program)
  = children in Head Start scored lower in academic achievement
  > what does this result mean? is the project a bad experiment?

Question 3

INUS condition - match example

Answer 3

• insufficient: a match cannot start a fire without the other conditions
• non-redundant: it adds something fireprompting that is uniquely different from what the other factors in the constellation (e.g. oxygen, leaves) contribute to starting the fire
• unnecessary: other factors can also start the fire; this constellation is not necessary
• sufficient: full constellation of factors is sufficient to start the fire

Question 4

what is the main feature of all experimental causes?

Answer 4

• they are manipulable
  > nonmanipulable events (explosion of a supernova) or attributes (age, gender, …) cannot be causes in experiments because we acnnot deliberately vary them to discover what happens
  ! only experimental causes must be manipulable, not all causes in general

Question 5

what is the cause in quasi experiments?

Answer 5

• whatever is being manipulated
  > the experiment may include many more things that were manipulated than what the researcher realizes
  > in quasi experiments, it’s particularly hard to see what is being manipulated and the context of manipulation

Question 6

Counterfactual & effect
- what are they?

Answer 6

• something that is contrary to the fact
  > in experiment, we observe what did happen
  > counterfactual, is what would have happened if those same poeple simultaneously had not received the treatment
  → effect = difference between what did happen and what would have happened

Question 7

counterfactual
- characteristics

Answer 7

• can never be observed
• it is impossible for a participant to simultaneously receive and not receive treatment
  → experiments try to create reasonable approximations to this physically impossible counterfactual
  → they also have to infer how experimental conditions differ from counterfactual

Question 8

• what experiment design gets close to counterfactuals?
• what are the problems with that?

Answer 8

• whithin-subject design
  > carryover, fatigue, practice, …
• random assignment and control groups
  > people in control group are not identical to treatment group
  > and in quasi-experiments, control group is systematic, not random
  → if the difference between cotrol and experimental condition would only rarely occur because of chance, we say that the differences are an effect of the treatment

Question 9

What is the first tool to estimate the counterfactual in quasi-experiments?
- in regards to Head Start project

Answer 9

1. measure academic achievement of kids way before project starts
2. estimate trajectories of academic achievement
3. measure academic achievement after project starts
4. compare measurements to estimations
   = you would see that Head Start boosted academic achievement

Question 10

What is the second tool to estimate counterfactuals in a quasi-experiment?
- in regards to the Head Start project

Answer 10

• try to make nonrandom control groups as similar as possible to the participants in the treatment group
  > Head Start matched kids in project to kids of same age, gender, grade, application to head start (even if not elegible)

Question 11

what is a problem with this matching tool?

Answer 11

• we don’t know all the variables that the children should match on
• controls in quasi-experiments are never as good as controls coming from randomization
• some of the variables that children were not matched on could be real cause of differences between post-test measurements between conditions

Question 12

what are possible solutions for the impracticability of these tools?

Answer 12

• time series statistics allow us to measure the rate of children’s academic maturation before project H.S., to then compare projection to actual result after project H.S.
  > if post-test exceeded projected score, then project H.S. was effective
• make nonrandom controls more similar to treatment by using statistics such as propensity score analysis or selection bias modeling, or by usign design features suhc as improved matching techniques
  ! both tools and solutions could be applied at same time; it is up to the researcher to make the quasi experiment as valid as possible

Question 13

Causal relationship
- how do we know if cause and effect are related?

Answer 13

(a) the cause preceded the effect
(b) the cause was related to the effect
(c) we can find no plausible alternative explanation for the effect other than the cause

Question 14

how can we measure consistency easily?

Answer 14

• statistical analysis
  > measure whether variation in the presumed cause is related to variation in the presumed effect
  > e.g. variation between treatment and outcome, or t test

Question 15

in what ways do quasi-experiments improve over correlational studies?
- priority criterion

Answer 15

• quasi-experiments force cause to precede effect by first manipulating the presumed cause and then observing outcome afterwards
  > e.g. if I make a person listen to jokes and measure mood later, I force one variable to precede the other

Question 16

in what ways do quasi-experiments improve over correlational studies?
- exclusivity criterion

Answer 16

• quasi-experiments allow researcher to control for some (but not all) of third-variable alternative explanations
  > e.g. I could control for siblings’ humor by studying singletons
  → if I still find relationship between mood and jokes, it cannot be because of siblings’ humor
  ! quasi-experiments can almost never control for all third variables because the researcher almost never knows what those third variables are

Question 17

what did campbell do?

Answer 17

• he codified some of the most commonly encountered group differences into a more general system of threats to valid causal infererence
  = he put together a list of common reasons why researchers make mistakes when inferring causality

Question 18

What are some threats to invalid causal inference?

Answer 18

• History
• Maturation
• Selection
• Attrition
• Instrumentation
• Testing
• Regression to the mean

Question 19

History
- what it is + examples

Answer 19

• events occurring concurrently with the treatment that could cause worse performance
  > e.g. children in H.S. project were moved into a classroom in a different hallway half-way through the year
  > e.g. high-profile hate crimes occurring in the neighborhood where H.S. was located

Question 20

Maturation
- what it is + examples

Answer 20

• naturally occurring changes over time
• could be confused with a treatment effect
  > e.g. children whose parents don’t read to them often develop vocabulary slower

Question 21

Selection
- what it is + examples

Answer 21

• systematic differences over conditions in respondent characteristics that could also cause the observed effect
  > e.g. criteria used to select children for H.S. project might lead to selecting more disadvantaged children (more poorly nourished etc → performance affected)

Question 22

Attrition
- what is it + examples

Answer 22

• loss of respondents to treatment can produce artifactual effects if that loss is systematically correlated with conditions
  > e.g. maybe most accomplished children in H.S. project got accepted to a private school, so the rest of the group performs worse than average

Question 23

Instrumentation
- what it is + examples

Answer 23

• the nature of a measure may change over time or conditions, affecting the results
• could be confused with treatment effect!
  > e.g. if raters change their rating standards they apply as a result of practice, this could result in lower scores for children over time

Question 24

Testing
- what it is + examples

Answer 24

• exposure to a test can affect scores on subsequent exposures to that test
• could be confused with treatment effect
  > e.g. discouragement after pre-test, motivation, fatigue, …

Question 25

Regression to the mean
- what it is + examples

Answer 25

• units with extreme scores will usually have less extreme scores on other variables
  > e.g. person that scores highest in a test is not likely to score highest in following test; tallest person is often not the heaviest
  > e.g. children that scored highest in H.S. exam were selected for project, but then scored lower the following time (seems like they did worse)
  ! it happens when two variables are not perfectly correlated with each other

Question 26

What are the characteristics of these threats to validity?

Answer 26

• they are endless
• not feasible to rule out all possible alternative explanations for effect, but we must focus on the plausible ones

Question 27

Why are plausibility judgements fallible?
= when judging whether alternative hypotheses are plausible, why are these judgements subject to mistakes?

Answer 27

• labeling rival hypotheses as plausible depends not just on what is logically possible, but also on social consensus, shared experience and empirical data
• it’s very hard to confirm a hypothesis, but easy to disprove it (Popper)
  = conclusions that withstand falsification are treated as plausible until shown otherwise

Question 28

can falsification be certain? why?

Answer 28

• it can never be certain
• it depends on two assumptions that cannot be fully tested

Question 29

What first assumption does falsification depend on?

Answer 29

• “causal claim is clear, complete and agreed upon in all its details”
  > this is never the case
  > many features of both claim and test of claim are debatable (e.g. operationalization, …)
  → often when data shows that hypothesis is wrong, researchers often make small adjustements to causal theory, without disregarding whole theory

Question 30

What second assumption does falsification depend on?

Answer 30

• falsification requires observational procedures that perfectly reflect the theory that is being tested
  > our observations are never that perfect
  > observations usually reflect only part of the question of interest, and they reflect rearcher’s wishes, hopes, aspirations, …

Question 31

Causal studies - final notes

Answer 31

• causal studies are useful when we have to change the initial hypothesis repeatedly to accomodate results
• tests can be trusted after being proved valid and reliable