Lecture 24 Flashcards
What do we mean by science is in a paradigm shift?
Science is in a paradigm shift
A paradigm is a model for scientific inquiry
Paradigm shift is a fundamental change in the current paradigm
Psychology has initiated this reform!
What did Bem’s study find? What was the design of the study?
Bem finds that his undergraduate students can see into the future, and change their present behavioru to maximize the benefits to them
Design: 9 within-subject exps, 1000+ participants
Method: Behind one curtain is an erotic image
Participant selects right or left curtain
Then, computer randomly assigns images
Result: Participants select porn curtain 53% of the time, a statistically significant difference from 50%
it is within subjects which accounts fully for participant variables which could otherwise confound our design. We could have order effects though.
large sample size, if there is an effect we are more likely to find it.
What were the failures to replicate Bem’s study?
Robinson, 2011
Ritchie et al., 2012
Galek et al., 2012
Traxler et al., 2012
Wagenmakers et al., 2012
Baruss & Rabier, 2014
only difference is that the researchers were sampline from a slightly different population and they were replicated around 6 to 12 months after the study
What is OSC? what is its goal? What was the method? What were the results?
open science collaboration
estimating reproducability
Method: Replicate 100 exp’s from 2008 § Top 3 Psychology Journals
Replications used original designs
Highly-powered (expect few Type II errors)
if there is a real effect we’ll be expecially likely to find that real effect
39% studies replicate, or were statistically significant
there were probably some studies where they got an unlikely streak of measurement error. What this means is likely around 50% of what we see in these top journals is not true.
What is another question that the OSC had? How did they research it?
Can we simulte false conclusions using currently acceptable methods?
they took hypotheses that they knew were false, used all the flexibility of the scientific method ansd tried to see if they could provie it
What was the simons 2011 experiment? What is the message?
Experiment 1:
IV: Song type (“Hot potato” vs. control)
DV: “How old do you feel right now?”
Result: “People felt older after listening to ’Hot Potato’
than after listening to the control song, p < .05.”
Discussion: People contrast their age against children’s song, mak
Experiment 2:
IV: Song type (“When I’m 64” vs. control)
DV: “What is your age?”
Result: “People were 1.5 years younger after listening to “When I’m 64” compared to the control song, p < .05”
Discussion: We found chronological rejuvenation!!
When reporting:
They only told us about 2 of the 4 conditions in both of the experiments because they showed positive results.
They stopped the study once finding statistical findings. If you’re analyzing your data at all intermediate points, what we’re doing at each analysis points is looking for random error.
Message: Relatively common, but unreported, scientific practices can find evidence for effects known to be wrong
What are the 3 points on the interim review
- Bem (2011) published seemingly impossible effect
- Many research teams fail to replicate Bem (2011)
- Open Science Collaboration (2015) estimates replication rates at about 40%
- Simmons (2011) shows that current paradigm can support incorrect hypotheses
How did we get to the point where we were replicating things we knew were false?
Criteria used to evaluate research: (This determined whether our experiment is published )
1. Finding answers important question
2. Finding has high internal validity
3. Finding is novel or surprising
4. Finding is statistically significant, p < .05
Filters for publication:
Significant & novel effects are published, even if…
Non-significant effects go to ‘file drawer”, even if….
Filters for scientists:
1. Publish to get a PhD
2. Publish to get a post-doc
3. Publish to become a prof
4. Publish to get grant funding
5. Publish to get tenure
“The more any quantitative social indicator is used for decision- making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor.”
~ Donald Campbell
Science is a social process & relies on human decisions!
Social indicator 1: Is the result statistically significant? Social indicator 2: Did you, scientist, publish?
the important thing is that the p values that are higher are relatively consistent but then as soon as the p value goes below 5%, you are folling off a cliff. So are people nudging their results in a specific direction?
What did the survey asking about scientific misconduct find?
In your career have you engaged in any of the following practices:
Not reporting all outcome variables: 66.5%
Collecting more data if not significant: 58.0%
Reporting only studies that ‘work’: 50. 0%
Not reporting all conditions: 27. 4%
Stopping data collection early: 22. 5%
“Rounding” p-values: 23. 3%
Falsifying data: 1.7%
What are some good things happening regarding replication and scietific honesty rn that may lead to a birghter future?
- Preregistering hypotheses, data collection, and data analysis plans
- Publishing null results (p > .05) and replications
- Increasing sample sizes
- Publishing raw data, methods, & analysis scripts
- Embracing ‘messy’ data
- Even (bravely) critiquing own work
What did Carney say about her research on power poses?
she siad too much research has not been able to replicate it and she believes that the effect is not real.
