Chapter 11- Confounding and obscuring variables

Question 1

One group, pretest/posttest design

Answer 1

A researcher recruits one group of participants, measures them on a pretest, exposes them to a treatment, intervention, or change, and measures them on a posttest. This design is problematic because there is only one group (no comparison group).

Question 2

6 potential internal validity threats in one group, pretest/posttest designs

Answer 2

1. Maturation threats
2. History threats to internal validity
3. Regression threats to internal validity
4. Attrition threats to internal validity
5. Testing threats to internal validity
6. Instrumentation threats to internal validity

Question 3

Maturation threat

Answer 3

A change in behavior that emerges more or less spontaneously over time. Kids might adapt to their summer camp over time and become more well behaved on their own.

Question 4

How are maturation threats prevented?

Answer 4

A true experiment, with a comparison group, would prevent maturation threats

Question 5

History threats

Answer 5

Result from an external factor that systematically affects most members of the treatment group at the same time as the treatment itself. This makes it unclear whether change was caused by the treatment received. Like if people used less energy because the weather got cooler, not because of a clean energy campaign

Question 6

How are history threats prevented?

Answer 6

A comparison group can also prevent history threats

Question 7

Regression threat

Answer 7

Refers to the statistical concept of regression to the mean. Regression threats only occur when a group is measured twice and has an extreme score at pretest. For example, depressed people seek treatment at their lowest, which could explain why they improved (by chance) after treatment. At posttest, it’s unlikely there would be the same combination of random unlucky factors.

Question 8

How are regression threats prevented?

Answer 8

Comparison groups and inspection of the pattern of results can prevent regression threats. If the comparison and experimental groups are equally extreme at pretest, researchers can account for any regression effects in their results.

Question 9

Regression to the mean

Answer 9

This means that when a group mean is unusually extreme at time 1, it will be less extreme (close to normal) at time 2. For example, you might normally be cheerful, but be in a bad mood one day due to various factors like traffic or the weather. It’s unlikely this combination of factors will happen again, and you will typically be back to normal tomorrow.

Question 10

Attrition threat

Answer 10

Attrition can happen when a pretest and posttest are administered on different days and some people drop out before the posttest. Affects internal validity when attrition is systematic- when only a certain kind of participant drops out. Ex- depression levels in a group might only appear to improve because the three most depressed participants dropped out.

Question 11

How are attrition threats prevented?

Answer 11

Usually, participants’ scores will be removed when they drop out. This prevents attrition threats.

Question 12

Testing threat

Answer 12

A specific kind of order effect, refers to a change in the participants as a result of taking a test (dependent measure) more than once. People might have practiced at taking the test, leading to improved scores, or may become fatigued or bored and get worse scores over time.

Question 13

How are testing threats prevented?

Answer 13

To avoid testing threats, the researchers might use a posttest only approach. They might use alternative forms of the pretest to measure different things. A comparison group is also helpful. If both groups take a pretest and posttest and the treatment group has a larger change, testing threats can be ruled out.

Question 14

Instrumentation threat

Answer 14

Occurs when a measuring instrument changes over time. Coders are measuring instruments, and they might change their standards over time. Different from testing threats, where it’s the participants that are changing, not the instruments.

Question 15

How are instrumentation threats prevented?

Answer 15

To prevent these threats, researchers might use a posttest only design. They might also collect data from each instrument to make sure they’re both the same. They might retrain coders over time.

Question 16

Selection-history threat

Answer 16

An outside event or factor affects only those at one level of the independent variable. For example, maybe a comparison group was only affected by the factor, but not the treatment group.

Question 17

Selection-attrition threat

Answer 17

Only one of the experimental groups experiences attrition. For example, in a depression study, the treatment might be most arduous for the treatment group, and the most depressed participants only drop out from the treatment group.

Question 18

3 potential internal validity threats in any study

Answer 18

1. Observer bias
2. Demand characteristics
3. Placebo effects

Question 19

Observer bias

Answer 19

Occurs when researchers’ expectations influence their interpretation of the results. They might expect the treatment to work and few the results more positively. Comparison groups might not necessarily help here

Question 20

Which validities are threatened by observer bias?

Answer 20

It threatens both internal and construct validity (ratings for variables are also inaccurate)

Question 21

Demand characteristics

Answer 21

This is a problem when participants guess what the study is supposed to be about and change their behavior in the expected direction. Participants might change their self reports to reflect this.

Question 22

How can demand characteristics be prevented?

Answer 22

To prevent this, researchers can use a double blind study- neither the participants nor the researchers who evaluate them know who is in the treatment group and who is in the control group.

Question 23

Masked design

Answer 23

Participants know which group they’re in, but the observers don’t.

Question 24

Placebo effect

Answer 24

Occurs when people receive a treatment and really improve- but only because the recipients believe they are receiving a valid treatment. This can occur with any type of treatment. Placebo effects are not imaginary- placebos reduce real symptoms and side effects, both psychological and physical

Question 25

Double blind placebo controlled study

Answer 25

One group gets the treatment, the other gets a placebo, and the study is double blind- prevents placebo effects.

Question 26

How can you show a placebo effect?

Answer 26

There can be lots of factors influencing improvement in the placebo group. To show a placebo effect, you would have to include a nontreatment comparison group- participants don’t get treatment or the placebo.

Question 27

With so many threats, are experiments still useful?

Answer 27

Responsible researchers consciously avoid internal validity threats when they design and interpret their work. Most of these threats are only a problem when no comparison group is included.

Question 28

Null effect

Answer 28

When a CI includes zero. In this case, the dependent variable didn’t show much change after manipulation of the independent variable. This might indicate something valuable- maybe the independent variable really doesn’t influence the dependent variable. It’s also possible that there is a true effect, but this particular study didn’t detect it. There can be slight variations in CI, and CIs might include zero by chance.

Question 29

What types of studies can have a null effect?

Answer 29

This can happen in posttest only designs, within groups designs, pretest/posttest design, and in correlational studies.

Question 30

2 obscuring factors that could take place in the study that cause a null effect

Answer 30

1. Perhaps there is not enough between groups difference

2. Perhaps within groups variability obscured the group differences

Question 31

What can cause less between groups difference? (4)

Answer 31

1. Weak manipulations
2. Insensitive measures
3. Ceiling and floor effects
4. Design confounds

Question 32

Weak manipulations

Answer 32

For example, giving people a small amount of money likely wouldn’t have a big effect on their mood. It’s important to ask about construct validity- how did the researchers operationalize the independent variable?

Question 33

Insensitive measures

Answer 33

Sometimes a study finds a null result because the researchers have not used an operationalization of the dependent variable with enough sensitivity. If a medication reduces temperature by .01 degree, a thermometer isn’t sensitive enough to detect it.

Question 34

Ceiling effect

Answer 34

Scores are squeezed together at the high end. Can be the result of a problematic independent or dependent variable. Ex- telling participants they are getting different levels of an electric shock to manipulate anxiety will likely cause a ceiling effect. Regardless of how high the shock, the participants will likely still be very anxious and anxiety scores cluster at the ceiling.

Question 35

Floor effect

Answer 35

All the scores cluster together at the low end, resulting from a problematic independent or dependent variable. Ex- a logical reasoning test that was so difficult that no one could solve the problems would cause a floor effect because everyone would get a very low score.

Question 36

How do manipulation checks detect ceiling and floor effects?

Answer 36

For the electric shock study- asking all participants to rate their level of anxiety beforehand would reveal the ceiling effect.

Question 37

How can design confounds cause less between groups variability?

Answer 37

Design confounds can actually reverse a true effect of an independent variable. For example, maybe students given more money had a rude examiner and students receiving less money had a cheerful examiner.

Question 38

Noise

Answer 38

Too much unsystematic variability in a group. This threatens statistical validity. There’s so much variability within groups that it obscures the differences between groups. The greater the overlap between members of the two groups, the less precisely the two group means are calculated and the smaller the standardized effect size.

Question 39

What happens to the CI with less variability within groups?

Answer 39

With less variability within the groups, the CI is narrower and the standardized effect size is larger.

Question 40

Measurement error

Answer 40

A human or instrument factor that can randomly inflate or deflate a person’s true score on the dependent variable. A person’s height might be influenced because they slouched as they were measured. All dependent variables involve a certain amount of measurement error, but researchers try to keep those errors as small as possible.

Question 41

Solutions for measurement error (2)

Answer 41

1. Use reliable, precise tools

2. Measure more instances

Question 42

Solutions for individual differences causing within groups variability (2)

Answer 42

1. Change the design

2. Add more participants

Question 43

Situation noise

Answer 43

External factors that could cause variability within groups and obscure true differences

Question 44

Power

Answer 44

An aspect of statistical validity- it’s the likelihood that a study will return an accurate result when the independent variable really has an effect.

Question 45

Studies with large samples have 2 major advantages

Answer 45

1. Large samples make the CI narrow- more precise estimates

2. Effects detected from small samples sometimes can’t be repeated