Chapter 14 - Generalizability

Question 1

college sophomore problem

Answer 1

• Most researchers are university profs -> do most research at universities -> most participants are university students (convenience sampling) -> the bulk of psychological knowledge is based on undergrads
• Solution: collect samples outside of college, and statistically analyze whether their responses are different (look for interactions)

Question 2

3 responses to college sophomore problem

Answer 2

1) Does not invalidate previous results – just need more findings
- Failure to replicate = ideas are incomplete, not wrong
- Why failure to replicate? How might college students and non-college students differ? (Boundary conditions)
2) A lot of research is on very basic processes
- People have similar basic processes (eg. Vision systems), regardless of whether they’re college students or not
3) College populations are actually very diverse spanning many income levels, ethnicities, ages, and other demographics

Question 3

2 different levels of generalizability

Answer 3

• context-based issues

- people-based issues

Question 4

context-based issues

Answer 4

• we want findings from our lab to generalize not only to other labs, but to the external world as well
• ex. other experimental settings, the real world

Question 5

people-based issues

Answer 5

• we want our results to generalize to other people (ex. genders, sexualities, age groups, etc.)
  ex. college sophomore problem, gender differences, cultural differences

Question 6

other experimental settings

Answer 6

• One experimental setting may not generalize to other experimental settings
• Interaction effects between participant and experimenter a big factor (ex. Biosocial effects, psychosocial effects)
• Problem when running in different labs
• Solution: standardize interaction between experimenters and participants

Question 7

the real world

Answer 7

• Studies that resemble the real world = high ecological validity
• can be achieved through mundane realism or experimental realism

Question 8

mundane realism

Answer 8

• when the situation resembles something that one would experience in everyday life
• Ex. Studies where aggression is measured by how much hot sauce people put into someone else’s drink are low on mundane realism
• Solution: create DV’s with high mundane realism

Question 9

experimental realism

Answer 9

• when the setting induces a realistic experience, even in an artificial setting
• Ex. Studies measuring aggression by using situations like hot sauce don’t’ really get students in the mindset that they would be in during actual aggressive situations using gun violence
• Solution: just use experimental realism

Question 10

how settings affect generalizability

Answer 10

• the more realistic the setting, the more generalizable
• ex. in lab settings:
• Sanitized version: telling participants about genetic/environmental explanations for a crime, then ask them how guilty they think a perpetrator would be
• Slightly less sanitized version: providing realistic case report with explanation of crime -> more generalizable
• ex. In a more realistic setting: doing a mock jury trial to see how they would react -> most generalizable

Question 11

gender differences

Answer 11

• Demonstrable differences between males and females (psychological literature on other sexes still lacking)
• Males and females may interpret variables differently or respond differently
• Solution: include both males and females as participants to analyze responses

Question 12

cultural differences

Answer 12

• People from different cultures have demonstrable differences
• Most psychology studies suffer from the WEIRD bias (<15% of world’s population is basis of theories about 100% of the world)
• Solution: collect data from other cultures & look for interactions

Question 13

Types of studies (other than single studies)

Answer 13

• triangulate studies
• replication studies
• meta-analyses
• mini meta-analyses
• publishing null results

Question 14

triangulate studies

Answer 14

• using more than 1 method to collect data on the same topic
• Study 1: relationship between belief about explanation for obesity, and beliefs about weight control
• Study 2: manipulating explanation for metabolism affects people’s beliefs about weight control
• Study 3: manipulating explanation for obesity, affects how much one eats in a study

Question 15

replication studies

Answer 15

• Can other researchers get the same effect?

- includes direct replication, conceptual representation, replication-plus-extension

Question 16

direct replication

Answer 16

• Key question: was original finding real, or was it by chance; new study exactly copies the methodologies and materials of the original studies
• Problem: new study may carry the same flaws (ie. 3rd variable problem); the effects you find may only be easy to find in your particular design

Question 17

conceptual representation

Answer 17

• Key question: what was the original study trying to examine; new study has new design and/or materials that are conceptually related to original study
• Problem: may promote Type 1 errors, not a substitute for direct replications

Question 18

replication-plus-extension

Answer 18

• Type of direct replication
• Key question: was the original finding real, and what else can I do with it? New study partially uses the same materials, but adds additional parts to address additional questions
• Problem: if anything about the initial experiment (like the IV) was weak/bad

Question 19

mini meta-analyses

Answer 19

• Researchers do multiple studies, then do meta-analysis on own studies in one paper
• This redirects attention away from p values and arbitrary thresholds
• Encourages inclusion of non-significant results, helps with file-drawer problem (when people tuck non-significant data away)
• Better estimates of effect size

Question 20

publishing null results

Answer 20

• Important to know that some things don’t lead to other things
• Journals that publish null results are rare and are less prestigious than ones that focus on significant results

Question 21

open science

Answer 21

• Movement of making knowledge accessible to everyone -> we should not be hiding science behind a paywall (journals charge a lot of money for people/institutions to access those journals)
• Open Science Foundation’s RRR -> encourages transparency by having researchers declare their hypothesis and planned analyses before they begin their study
• Incentives: Psychological Science’s Open Badges -> open data, open materials, pre-registration

Question 22

what does failed replication mean?

Answer 22

A failed replication may not mean the original study was wrong, it could mean the replication was flawed

Question 23

effect size

Answer 23

• Independent of sample size

- Focuses on the size of the difference between groups rather than p value

Question 24

WEIRD bias

Answer 24

• Western
• Educated
• Industrialized
• Rich
• Democratic
• unable to generalize to all humanity