Exam 2 Flashcards
1
Q
for a study to be an experiment, it has to have…
A
- at least one manipulated variable
- at least one measured variable
2
Q
control variable
A
a variable that an experimenter holds constant on purpose (besides the independent variables)
- not really variables because they do not vary
3
Q
why experiments support causal claims through the 3 criteria
A
- establish covariance: changes in the independent variables are related to the changes in the dependent variable
- establish temporal precedence: The causal variable should come before the outcome variable
- establish internal validity: There are no other likely explanations for the relationship observed
4
Q
placebo (control) groups
A
a group that is exposed to an inert treatment; comparison group may not need to be a control group (i.e., no treatment)
5
Q
confounds
A
an unmeasured variable that influences both the supposed cause and the supposed effect
6
Q
design confounds
A
an accidental second variable varies systematically along with the intended independent variable
7
Q
selection confounds (selection effects)
A
when the kinds of participants in one level of the independent variable are systematically different from those in the other
- avoid selection effects with random assignment
- avoiding selection effects with matched groups
8
Q
types of experimental design
A
- independent-groups designs (between-sujects)
- within-subjects designs
- pottest-only designs
- pretest/posttest designs
9
Q
independent-groups designs (between-subjects)
A
Separate groups of participants are placed into different levels of the independent variable
- Ex. an experiment exploring how different amounts of sleep affect people’s reaction times → level 1: 3 hours of sleep, level 2: 8 hours of sleep
10
Q
within-subjects designs
A
- Each person is presented with all levels of the independent variable
- One set of participants are tested more than once, and their scores are compared
- Ex. an experiment exploring how different amounts of sleep affect people’s reaction times → all groups do day 1 with 3 hours of sleep and day 2 with 8 hours of sleep
11
Q
posttest-only designs
A
Participants are randomly assigned to independent variable groups and are tested on the dependent variable once
12
Q
pretest/posttest designs
A
Participants are randomly assigned to at least two difference groups and are tested on the key dependent variable twice— once before and once after exposure to the independent variable
13
Q
problems using pretest/posttest design
A
threat to internal validity
- Taking the pretest affects how participants do the posttest → testing threat
- Participants may get tired from a long study with a pretest and posttest
14
Q
repeated-measures design
A
participants are measured on a dependent variable more than once, after exposure to each level of the independent variable
15
Q
concurrent-measures design
A
participants are exposed to all the levels of an independent variable at roughly the same time, and a single attitudinal or behavioral preference is the dependent variable
16
Q
advantages of within-groups design
A
- Participants in your groups are equivalent because they are the same participants and serve as their own controls
- Gives researchers more power to notice difference between conditions because there is less extraneous error in the measurement
- Requires fewer participants
17
Q
disadvantages of within-groups design
A
- potential for order effects
- carryover effects
- might not be practical or possible
- experiencing all levels of the IV changes the way participants act –> demand characteristics
18
Q
demand characteristics
A
subtle cues or aspects of an experiment that might unintentionally signal to participants what the study is about, leading them to change their behavior to fit that perceived expectation
19
Q
order effects
A
When the sequence in which stimuli are presented to participants influences their responses; order of conditions can affect the results
when being exposed to one condition affects how participants respond to other conditions
20
Q
carryover effects
A
- practice effect –> participants perform better during later treatment conditions because they’ve had time to practice and improve
- Fatigue effect → participants perform worse during later treatment conditions because they’re tired or fatigued
21
Q
avoiding order effects
A
- full counterbalancing
- partial counterbalancing
22
Q
solution to order effects
A
counterbalancing
23
Q
interrogating causal claims with construct validity
A
How well were the variables measured and manipulated?
- dependent variables: check face validity, interrater reliability, and convergent vailidty
- independent variables: How well were they manipulated?
- manipulation check
24
Q
interrogating causal claims with external validity
A
To whom or what can the causal claim generalize?
- generalizing to other people
- generalizing to other situations
25
interrogating causal claims with statistical validity
How well do the data support the causal claim?
- accuracy of the conclusions drawn from a study's statistical analysis
- Is the difference statistically significant?: P-value < .05 usually considered statistically significant
- How large is the effect?: correlation coefficient (r), Cohen's d
- confidence interval
26
interrogating causal claims with internal validity
Are there alternative explanations for the outcome?
- Were there any design confounds?
- If an independent-groups design was used, did they control for selection effects using random assignment or matching?
- If a within-groups design was used, did they control for order effects by counterbalancing?
27
manipulation check
extra dependent variable that researchers can include to convince them that their experimental manipulation worked
ex. A study comparing the effect of a serious lecture vs. a funny lecture on the memory of lecture information; Manipulation check - how funny was the lecture?
28
correlation coefficient (r)
indicates the strength of a linear association between two variables (association claim)
29
Cohen's d
standardized effect size for measuring the difference between two group means
(group A mean - group B mean) / pooled standard deviation
- small (d = 0.2)
- medium (d = 0.5)
- large (d = 0.8)
30
confidence interval
a range of values, bounded above and below the statistics's mean, that likely would contain an unknown population parameter
- when a study has a small sample and more variability, CI will be relatively wide (less precise)
- when a study has a larger sample and less variability, CI will be narrower (more precise)
31
six potential internal validity threats in one-group, pretest/posttest designs
- maturation
- history
- regression
- attrition
- testing
- instrumentation
32
maturation threat to internal validity
A change in behavior that emerges more or less spontaneously over time
- Spontaneous remission is a specific type of maturation
- E.g., people adapt to changed environments; children get better at walking, talking, reading, etc.; plants grow taller
33
preventing maturation threats
Include an appropriate comparison group
34
history threats to internal validity
- Something specific has happened between the pretest and posttest (not just time has passed)
- “Historical” or external factor that systematically affects most members of the treatment group at the same time as the treatment itself
- E.g., the rowdy boys started a swimming course and the exercise tired most of them out
35
preventing history threats
include a comparison group
36
regression threats to internal validity
- regression to the mean: the tendency of results that are extreme by chance on first measurement to move closer to the average when measured a second time
- Occurs only when a group is measured twice, and
- Only when the group has an extreme score at pretest
- E.g., the 40 depressed women might have scores exceptionally high on the depression pretest due to random effects, such as recent illness, family or relationship problems
37
preventing regression threats
include a comparison group
38
attrition threats to internal validity
- A reduction in participant numbers that occurs when people drop out before the end of the study
- Problem for internal validity when attrition is systematic— only a certain kind of participant drops out
39
preventing attrition threats
Remove the dropped-out participants’ scores from the pretest average too
40
testing threats to internal validity
A change in the participants as a result of taking a test (dependent measure) more than once
41
preventing testing threats
- No pretest
- Two different forms— one for pretest and one for posttest
- Include a comparison group
42
instrumentation threats to internal validity
- Occur when a measure instrument changes over time
OR
- When a researcher uses different forms for the pretest and posttest, but the two forms are not sufficiently equivalent
- E.g., people judging the rowdy campers’ behavior became more tolerant of loud voices and rough-and-tumble play
43
preventing instrumentation threats
- Use a posttest-only design
- Ensure that the pretest and posttest measures are equivalent
- Counterbalance the versions of the test
44
three potential internal validity threats in any study
- observer bias
- demand characteristics
- placebo effects
45
observer bias
when researchers’ expectation influence their interpretation of the results
46
placebo effects
eople’s behaviors or symptoms respond not just to the treatment, but also to their belief in what the treatment can do to alter their situation
47
controlling for observer bias and demand characteristics
- double-blind study
- masked design (or blind design)
48
double-blind study
Neither the participants nor the researchers who evaluate them
know who is in the treatment group and who is in the comparison group
49
masked design (or blind design)
the investigator does not know the identity of the treatment assignment
50
preventing placebo effects
Use a double-blind placebo control study:
- One group receives real treatment, and another group receives the placebo treatment
- Neither the participants nor the investigators know who is in the experimental group or in the placebo group
51
interrogating null effects
1) the independent variable has no effect on the dependent variable
2) other reasons: a) not enough difference between groups, b) there is too much variability within groups
3) insensitive measures
52
not enough between-groups difference in interrogating null effects
- Weak manipulations
- Insensitive measures
- Ceiling and floor effects
- Reverse design confounds
53
ceiling and floor effects
Both control and experimental groups scored very high or very low
54
reverse design confounds
Design confounds acting in reverse
55
weak manipulations
ask: How did the researchers operationalize the independent variable?
E.g., Does money make people happy?
- A researcher gives one group of participants $1 and another group $2
- The manipulation is not strong → try $1 vs. $50
56
insensitive measures
- Dependence measure was not sensitive enough
- Solution: use a detailed, quantitative increments measures
57
solution to ceiling and floor effects and dependent variables
use a manipulation check
58
within-groups variability in interrogating null effects
Too much unsystematic variability within each group → noise (error variance or unsystematic variance)
59
measurement error in interrogating null effects
- Error in the measurement
- All dependent variables involve a certain amount of measurement error
- Solution 1: use reliable, precise tools; Have excellent reliability (internal, interrater, and test-retest)
- Solution 2: measure more instances
60
solution to individual differences in interrogating null effects
- Solution 1: change the design; Use a within-groups design instead of independent-groups design
- Solution 2: add more participants
61
solution for situation noise in interrogating null effects
carefully control the surroundings of an experiment
62
power
the likelihood that a study will return an accurate result when the independent variable really has an effect
63
what increases power
- Within-groups design
- A strong manipulation
- A larger number of participants
- Less situation noise
64
experiments with two independent variables can show interactions
To test whether the effect of the original independent variable depends on the level of another independent variable
- Interaction = a difference in differences = the effect of one independent variable depends on the level of the other independent variable
65
factorial design
when there are two or more independent variables (or factors)
- can test limits
- can test theories
66
factorial design and testing limits
- a form of external validity (testing the generalizability)
- interactions show moderators
67
moderator
a variable that changes the relationship between two other variables
68
interpreting factorial results
- main effects
- interactions
- If you have two independent variables, there will be two main effects and one possible interaction between them
- With three independent variables, there will be three main effects and several possible interactions (including two-way interactions and a three-way interaction)
69
factorial variations
- Independent-groups factorial designs
- Within-groups factorial designs
- Mixed factorial designs
- Increasing the number of levels of an independent variable
- Increasing the number of independent variables
70
independent-groups factorial design
2 x 2 factorial design = 4 conditions
- different people in each group
71
within-groups factorial design
2 x 2 factorial design = 4 conditions
- One group of participants participate in all four combinations
- Has more statistical power than an independent-groups factorial design
72
mixed factorial design
One independent variable is manipulated as independent-groups (between-groups) and the other is manipulated as within-groups
E.g., cell phone use (on the phone vs. not on the phone) x age (old vs. young)
- Old participants take part in both “on the phone” and “not on the phone” conditions
- Same for the young participants
73
increasing the number of levels of an independent variable for factorial variation
- Simplest factorial design 2 x 2: Two independent variables with two levels in each independent variable
- Can add one or more levels in one or both independent variables: E..g, two independent variables with three levels in one of the independent variables
74
increasing the number of independent variables for factorial variation
E.g., 2 x 2 x 2 factorial, or a three-way design
- Three main effects
- Three possible two-way interactions
- One three-way interaction
75
quasi-experiments
- Do not have full experimental control
- First select an independent variable and a dependent variable
- Random assignment might not be possible
76
nonequivalent control group pretest/posttest design
ex. study investigating the psychological effects of cosmetic surgery
- group of people who got the surgery vs. group of people who registered at the same clinic but did not receive any procedure
77
nonequivalent control group posttest-only design
ex. opt-in vs. opt-out default options of organ donation across countries
- no control over which countries had which defaults
- no random assignments of people
78
interrupted time-series design
ex. investigating popular shows and suicide
- the variable (of the suicide rates in the US) was measured repeatedly— before, during, and after the "interruption" caused by some event (the introduction of the show 13 Reasons Why)
79
nonequivalent control group interrupted time-series design
ex. investigating the effect of legislation on opioid abuse
- Florida passed laws that medical clinics could not dispense opioids, North Carolina did not
- nonequivalent control group → not randomly assigned to having the pill mill laws or not
- Interrupted time-series → researchers did not have experimental control over the year the laws were passed
80
threats to internal validity in quasi-experiments
- selection effects
- design confounds
- maturation threats
- history threats
- repression to the mean
- attrition threats
- testing threats
- instrumentation threats
- observer bias, demand characteristics, and placebo effects
81
selection effects in quasi-experiments
- Relevant only for independent-groups designs, not for repeated-measures designs
- Applies when the kinds of participants at one level of the independent variable are systematically different from those at the other level
Example 1: nudging people toward organ donation
- No selection effect
Example 2: the psychological effects of cosmetic surgery
- Maybe selection effect, but not likely
82
design confounds in quasi-experiments
- Some outside variable accidentally and systematically varies with the levels of the targeted independent variable
Example 1: nudging people toward organ donation
- Maybe some other government policy co-occur with the presumed consent
- Not likely as all seven countries with presumed consent policies should also have the same or similar policy
83
maturation threats in quasi-experiments
In an experiment or quasi-experimental design with a pretest and posttest, the observed change could have emerged more or less spontaneously over time
Example 2: the psychological effects of cosmetic surgery
- The comparison group did not improve over time —> no maturation threat
Example 4: investigating the effect of legislation on opioid abuse
- The comparison group (North Carolina) overdose deaths did not decline —> no maturation threat
84
history threats in quasi-experiments
Occurs when an external, historical event happens for everyone in a study at the same time as the treatment
Example 3: popular shows and suicide
- Suicide rates might have increased because of a suicide of a celebrity
Example 4: investigating the effect of legislation on opioid abuse
- The results might have been caused by some change in employment or living conditions
85
regression to the mean in quasi-experiments
Only for pretest/posttest designs
Example 2: the psychological effects of cosmetic surgery
86
attrition threats in quasi-experiments
- In a pretest/posttest design, attrition occurs when people drop out of a study over time
- Becomes a threat to internal validity when systematic kinds of people drop out of a study
87
testing threats in quasi-experiments
A kind of order effect in which participants change as a result of having been tested before
Repeated testing:
- Might cause people to improve, regardless of the treatment they received
- Might also cause performance to decline because of fatigue or boredom
Example 2: the psychological effects of cosmetic surgery
The surgery group improved over time while the comparison group declined → no testing threat
88
instrumentation threats in quasi-experiments
- A measurement could change over repeated uses, and this can be a threat to internal validity
- Having a comparison group helps detect instrumentation threats if there are any
89
why choose a quasi-experiment
- Real-world opportunities
- External validity: Quasi-experiments capitalize on real-world situations, even as they give up some control over internal validity
- Ethics: A researcher might choose a quasi-experimental design when the questions they have would be unethical to study in a true experiment
- Construct validity: Quasi-experiments show excellent construct validity for the quasi-independent variable
90
quasi-experiments and correlational studies
In quasi-experiments, the researchers tend to select their samples more intentionally than correlational studies
91
quasi-independent variables compared with participant variables
Quasi-independent variables focus less on individual differences and more on potential interventions such as laws, media exposure, or education
92
participant variable
a categorical variable, e.g., age, gender, ethnicity
93
small-n designs
Obtain a lot of information from just a few cases instead of a little information from a larger sample
94
disadvantages of small-n studies
Issues with internal validity
Issues with external validity
- Participants in small-n studies may not represent the general population very well
- Solution: compare a case study results to research using other methods
95
behavior-change studies in applied settings: three small-n designs
- stable-baseline design
- multiple-baseline design
- reversal design
96
stable-baseline design
a researcher observes behavior for an extended baseline period before beginning a treatment of other intervention
97
reversal design
Behavior high during baseline sessions and lower during treatment sessions
98
quality of science
replication, transparency, applicability to real-world context
99
replicable (or reproducible)
- part of interrogating statistical validity: size of the estimate (effect size), precision of estimate (95% CI)
- gives a study credibility
100
types of replication
- direct replication
- conceptual replication
- replication-plus-extension
101
direct replication
repeat an original study as closely as possible
102
concerns with direct replication
- threats to internal validity or flaws in construct validity in the original study would be repeated
- when successful, it confirms what we already learned
- does not test the theory in a new context
103
conceptual replication
same research question as the original study, but use different procedures
104
replication-plus-extension
replicate the original experiment and add variables to test additional questions
105
Why might a study not be replicable?
- issues with the replication study itself (differences in sample, materials, or geography)
- issues with the original studies
106
meta-analysis
mathematically averaging the results of all the studies (both published and unpublished) that have tested the same variables
107
strengths of meta-analysis
- can sort the studies into categories --> can identify new patterns in the literature --> could lead to new questions to investigate
- solves the File Drawer Problem by including both published and unpublished data
108
File Drawer Problem
studies that were never published could have important insights and add to the effect size
109
questionable research practices
- underreporting null findings
- HARKing
- p-HACKing
110
underreporting null findings
reporting only strong effects, not the weak ones
111
HARKing
Hypothesizing After the Results are Known
- misleads readers about the strength of the evidence
112
p-HACKing
running different types of statistics to find p < 0.5
113
transparent research practices
- open data
- open materials
- preregistration
114
open data
others can analyze the data
115
open materials
others can replicate the study
116
preregistration
scientists publish their study's method, hypotheses, or statistical analyses before collecting data
117
ecological validity
extent to which a study's task and manipulations are similar to the kinds of situations participants might encounter in their everyday lives
- one aspect of external validity
