chapter 11

Question 1

What is the really really really bad experiment? Why is it known for that name?

Answer 1

It’s the on-group pretest-posttest design. It is known to be bad because of its 6 threats to internal validity :
1. Maturation threats
2. History threats
3. Regression threats
4. Attrition
5. Testing threats
6. Instrumentation threats
7. (Bonus) Combined threats
a. Selection-history threats
b. Selection-attrition threats

Question 2

What is a maturation threat? How can they be prevented? Give an example.

Answer 2

A maturation threat happens when an observed change in a group could have emerged spontaneously over time (because of natural development.
They can be prevented with a comparison group.
Example : Disruptive boys at summer camp get calmer by the end of the week because they’re tired.

Question 3

What is a history threat? How can they be prevented? Give an example

Answer 3

A history threat happens when it is unclear whether a change in the treatment group is caused by the treatment itself or an external or historical factor.
They can be prevented with a comparison group.
Example : Measuring depression in November 2019 vs. March 2020

Question 4

What is a regression threat? How can they be prevented? Give an example?

Answer 4

An experimental group whose average is extremely low (or high) at pretest will get better (or worse) over time because the random events that caused the extreme pretest scores do not recur the same way at posttest.
They can be prevented with a comparison group.
Example : A group’s average is extremely depressed at pretest, in part because some members volunteered for therapy when they were feeling much more depressed than usual.

Question 5

What is an attrition threat? How can they prevented? Give an example.

Answer 5

An attrition threat occurs when a systematic type of participant drops out of the study before it ends, affecting the mean (representing a dropout rate).
They can be prevented by removing participants from the first portion of the study.
Example :Because the most rambunctious boy in the cabin leaves camp early, his unruly behavior affects the pretest mean but not the posttest mean.

Question 6

What is a testing threat? How can they be prevented? Give an example.

Answer 6

A testing threat is a type of order effect; An experimental group changes over time because repeated testing has affected the participants. Practice effects (fatigue effects) are one subtype.
They can be prevented by abandoning the pretest.
Example : GRE verbal scores improve only because students take the same version of the test both times and therefore are more practiced at posttest.

Question 7

What is a an instrumentation threat? How can they be prevented? Give an example?

Answer 7

An instrumentation threat occurs when the tools that we’re using to collect the data changes over time, thus the experimental group “changes over time”.
They can be prevented by switching to a posttest design only and ensuring that the measures are consistent.
Example : Coders get more lenient over time, so the same behavior is coded as less disruptive at posttest than at pretest.

Question 8

What are the two combined threats? Give an example for each

Answer 8

1. Selection-history threat : Occurs when an outside event or factor systematically affects participants at one level of the IV, affecting ONLY one group.
   Example : Flood at Carleton and none at OttawaU
2. Selection-attrition threat : Participants in only one group experience attrition.
   Example : CarletonU students dropout but not those at OttawaU

Question 9

What are the three potential internal validity threats in ANY study? Give their definitions.

Answer 9

1. Observer bias : Occurs when observer expectations influence the interpretation of participant behaviours or the outcome of the study.
2. Demand characteristics : A cue that leads participants to guess a study’s hypothesis or goals.
3. Placebo effects : A response or effect that occurs when people receiving an experimental treatment experience a change only because they believe they are receiving a valid treatment.

Question 10

What is a null effect? Why may they occur?

Answer 10

A null effect occurs when we run an experiment with 2 levels of the independent variable, but there’s no covariance, no difference or no effect between the 2 groups.
The reason for null effects vary; the strength of the manipulation may not be very strong, the way in which scores are captured or measured may affect it, and the way we manipulated certain variables as well.

Question 11

What is a ceiling effect?

Answer 11

A ceiling effect is an experimental design problem in which independent variable groups score almost the same on a dependent variable, such that all the scores fall at the high end of their possible distribution.
Example : The questions were too easy, everyone gets them right.

Question 12

What is a floor effect?

Answer 12

A floor effect is an experimental design problem in which independent variable groups score almost the same on a dependent variable, such that all the scores fall at the low end of the distribution.
Example : The questions were too hard, everyone gets them wrong.

Question 13

What is a manipulation check?

Answer 13

A manipulation check is an extra dependent variable to determine how well a manipulation worked.

Question 14

What are some of the issues that can occur in a between-groups desing?

Answer 14

1. Measurement error
2. Situation noise
3. Individual differences

Question 15

What is a measurement error? Give an example. How can we reduce the probability of measurement error?

Answer 15

A measurement error is a human or instrument factor that can randomly inflate or deflate a person’s true score on the dependent variable.
Example : a person who is 160cm tall might be measured at 160.25 cm because of the angle of vision of the person using the meter stick, or they might be recorded as 159.75 cm because they slouched a bit.
Alt. Example : Measuring symptoms of depression with a measure which included other factors.
They can be prevented by using reliable, precise measurements as well as measuring more than once.

Question 16

How can individual differences affect independent-groups (between-subjects) variability? How can it be prevented? Give an example.

Answer 16

The fact that there are so many individual differences obscures the difference between the 2 groups.
They can be prevented by changing the design to a within-groups or matched-groups design ; we can also add more participants to increase sample size.
Example : People have varying moods regarding money, they may react different differently regarding the amount they receive.

Question 17

What is situation noise? How can it be prevented? Give an example.

Answer 17

Situation noise is any kind of external distraction that could cause variability within groups that obscures between-groups differences. It is a third factor that could cause variability within groups and obscure true group differences.
It can be minimized by controlling the surroundings of an experiment.
Example : Construction outside which can distract participants.

Question 18

What is the opposite of obscurring?

Answer 18

Power and precision.

Question 19

Give the definition to power.

Answer 19

Power is the likelihood that a study will yield a statistically significant result when the IV really has an effect. It is related to the size of the sample (the larger it is, the more power we have) and the effect size.

Question 20

True or False : Null effects are reported just as often as significant differences in research literature.

Answer 20

False : Null effects may be less likely to be reported in the popular media than other results.