Lecture 2: Doing psychological science, replication issues Flashcards
Name an advantage and disadvantage for each of the correlational approaches and the experimental approach
Correlational adv: shows how things are associated, more naturalistic (correlate things as they come)
Correlational disadv: correlational never equals causation
Experimental adv: it can show causal direction with more confidence
Experimental disadv: little ambiguity with potential confounds, less naturalistic because of manipulations, shows one causal direction but doesn’t necessarily see both directions
What is one way to make p-values more consistent across repeated studies (replications)?
Increase the sample size
If we measure things with more precision, use a “better” method (better experiments)
What is the difference between an exact replication and a conceptual replication?
Exact - using someone else’s study and doing the exact same methods as them to see if it will replicate
Conceptual - interested in the same idea but strategically changing the method . Aimed at telling us if the underlying idea is correct or if it is telling us things more generally.
What are some reasons (psychology) studies fail to replicate?
Good: actual phenomena has changed (1st study found a result in 1960 but contexts have changed so the results aren’t the same), falsifiability still important
Problematic: having multiple unreported variables (can’t take their findings for face value because there could be other things influenced this), cherry picking, results are due to chance, mistake or bias in replication attempt
What are “questionable research practices”? Provide a general explanation specific examples. What is the net effect of doing all of these in %?
Choosing measures that aren’t as significant as they should be but making them look significant
Examples: Multiple unreported dependent variables, adding statistical controls, adding participants, dropping experimental conditions
p<0.5 , 60%
What are some ways we can/are improving replicability in psychology?
Larger studies, investing more in a particular study
Multi-site collaborations
Creating rewards for replications (more rep studies published means more effort put into these studies)
What are the factors leading to the replication crisis?
What study really kickstarted the crisis?
Long history of gossip, questions about p-values, unreliability with small samples
2011 Multi-study paper on ESP by Daryl Bem, high-profile fraud
What is significant about the musical contrast and chronological rejuvenation study? what occurred in the study?
Undergrads who listened to “when I’m sixty-four” were nearly a year and a half younger afterwards than after listening to “Kalimba”
Two studies were conducted with real participants and employed legitimate stat analyses and were reported truthfully yet they seem to show impossible results - reveal potential issues with the way psychological studies are conducted
What incentivizes researchers to conduct potentially wonky studies?
Pressures to publish (publication bias) More is better New/novel is better Faster is better Null results not welcome Replications seen as "boring" Jobs, status and funding are at stake
What are the steps taken in the reproducibility project (2015)? What are the results?
Select representative sample of studies Pair new study with new researcher Consult original researcher for details Publicly record detailed plans Collect and analyze new data and results
About 1/3-1/2 or studies found the original results
Effect sizes were about 50% of original reports
Cognitive psych appeared to be more replicable than social psych
Viewed as disappointing
Why are studies failing to replicate potentially a positive thing?
Quantifies/estimates reproducibility rate
Suggests value in reform
Motivates better practices
Is very “scientific”
What are some ways forward from the replication crisis? Possible solutions?
More cautious view of published findings
Test potential moderators in new studies (discover what the results “depend on”)
More open science practices (pre-registering studies - makes methods, materials, procedures and data available)
Registered reports - published because of idea/blind to the results
Better methods: systematic research (ex: larger sample sizes)
Rewarding replication efforts (funding, prizes, status)
