Replication, Generalizability, and the Real World Flashcards
What was the controversial (kinda stupid) study conducted by Bem?
Feeling the future
Study was on “reverse priming” - participants first did a memory test and then studied the material
Bem claimed that studying has a backward effect in time to allow participants to perform better - the effect was stat sig (those who were primed after the test did better before)
Participants “sensed the future” - proof of psi
Refresher: what are the three tenets of a good theory?
- Falsifiable - it can be disproved
- Supported by the data - empirical evidence backs it
- Parsimonious - no simpler explanation
What was the result of Bem’s ESP article?
Triggered widespread controversy on whether research in psychology can be replicated - replication crisis
Also is other forms of psychology - dead fish in the FMRI
What is a direct replication study?
Repeating an original study as closely as possible to see whether the effect is the same in the newly collected data
ex: distraction study - replicate would use the exact same distraction
What is a conceptual replication study?
Researchers explore the same research question but use different procedures
ex: Distraction study - same concept but different distraction
What is a replication + extension study?
Researchers replicate the original experiment and add variables to test additional questions - makes the results more robust
ex: distraction study - is listening to your favourite music more, less or equally distraction
When the open science collaboration attempted to replicate 100 psych exp from top tier journals, what percent directly replicated?
36% directly replicated their og study
Why might a study fail to replicate or even attempt to be replicated?
- Important aspects of the og study could be changed or left out (ex: email vs. real mail)
- There are problems with the og study: small n sizes, questionable statistical or research practices etc.
- Problems with publication practices: not encouraged/ignored by journals, new and exciting findings are prioritized
Has Bem’s weird study been replicated? if so, what were the results?
Yes, with no evidence of extrasensory perception
What is HARKing? what does this attempt to do?
Hypothesizing after the results are known - researchers first find a significant result then say that they hypoT that result all along
Attempts to make exploratory research look like confirmatory research
- exploratory: generates new questions and ideas, looks for patterns
- confirmatory: tests specific hypoT
What is P-hacking? what does it involve?
Researcher runs multiple tests until they find a significant result - may involve running multiple group comparisons or massive correlation tables Data massaging (torturing): transforming or altering data until it reveals a significant result
Why do researchers hark and p-hack?
Publish or perish attitude in academia forces researchers to make it in science and publish in high impact journals
What is open science? what can it include as a defence against p-hacking and HARKing?
Practice of sharing data, procedures, and results freely so that others can verify the results
Includes pre-registration: researchers sharing all of their hypoT, methods, and proposed analyses before running the study - then they follow this plan
What is meta-analysis?
A way of combining all the studies on a topic and running one big statistical analysis to see what the overall effect is - synthesizing a large body of research
“Garbage in, garbage out” - if the studies are poorly conducted, the info will be meaningless
What are the two questions relating 2 generalizability?
Do the conclusions apply to
a) other people - from different age groups, ses, cultures
b) other situations - is the effect limited to a lavatory environment does time of day matter
