Lecture 11: Critical Thinking About Psychological Research Flashcards
What are the 4 problems with analyzing and reporting data and explain them
- Replicability - failure to repeat the same experiment with the same findings (different data set, same procedure)
- Robustness - conclusions that authors draw are not the same as the conclusions other draw (same data, different procedure)
- Reproducibility - different conclusions are drawn even when data and procedure are the same (same data, same procedure)
- Flexibility-ambiguity problem - ambiguity = how to make the best decisions when handling data, flexibility = researchers often stop data collection when results are significant
What is the researchers degrees of freedom
The choices researchers can make in how they analyze results and what they report
What are 6 requirements for authors
- Authors must decide the rule for terminating data collection before data collection begins and report this rule in the article
- Authors must collect at least 20 observations per cell or else provide a compelling cost-of-data collection justification
- Authors must list all variables collected in a study
- Authors must report all experimental conditions, including failed manipulations
- If observations are eliminated, authors must also report what the statistical results are if those observations are included
- If an analysis includes a covariate, authors must report the statistical results of the analysis without the covariate
What are 4 guidelines for reviewers
- Reviewers should ensure that authors follow the requirements
- Reviewers should be more tolerant of imperfections in results
- Reviewers should require authors to demonstrate that their results do not hinge on arbitrary analytic decisions
- If justifications of data collection or analysis are not compelling, reviewers should require the authors to conduct an exact replication
What are 4 guidelines for reviewers
- Reviewers should ensure that authors follow the requirements
- Reviewers should be more tolerant of imperfections in results
- Reviewers should require authors to demonstrate that their results do not hinge on arbitrary analytic decisions
- If justifications of data collection or analysis are not compelling, reviewers should require the authors to conduct an exact replication
What are 4 types of biases and explain them
- Asymmetric attention = motivated reasoning, people tend to process information in such a way that it matches the goals they want to achieve
- Hypothesis myopia = focused on things that align with your initial beliefs - try and look for evidence that would support our hypothesis, ignoring alternative interpretations
- Texas sharpshooter = seeing a random pattern in the data and declaring it as meaningful
- Just-so storytelling = when you believe something is more likely after the fact because you’ve already seen it happen
What are 3 solutions to biases and explain them
- Transparency = open science and preregistering
- Team of rivals = invite academic rivals to work with you and try to criticize your work
- Blind data analysis = researcher handles the data first before getting to see the results of the analysis
What is HARKing
HARKing is defined as developing or changing hypotheses based on the results of a study, often after data analysis has been conducted, rather than forming hypotheses before data collection
What is p-hacking
This occurs when researchers collect or select data or statistical analyses until nonsignificant results become significant.
What is Actively Open-minded Thinking
Looking for alternative explanations for a certain effect - being critical
What happens to the power if the sample size of a study is small
Low power = more difficult to detect small effects
T/F: the probability of making a type 1 error on each individual test is 0.05, but the probability of making a type 1 error for the whole collection of tests is higher
True
What are 4 publication and reporting biases that influence which and the way articles are published and what do they mean
- Study publication bias: studies that don’t find significant results are much less likely to get published
- Outcome reporting bias: some studies that didn’t find significant results engage in questionable research practices to be able to report significance
- Spin: some authors write their results in a way that makes it seem like the results were significant, without specifically stating that but making it look like they were
- Citation bias: studies that found a significant effect are more likely to be cited in other studies
What are the two phases of AOT and explain them
- Search phase - search for possibilities, your own hypothesis but also alternative explanations for the phenomenon
- Inference phase - be thorough and fair in inference; look at evidence and relate to the possibilities
What are 5 issues in psychological research and explain them if necessary
- Too much emphasis on new, surprising findings
- Too much emphasis on counterintuitive findings (explain)
- Smalll and noisy samples (explain)
- Popular media will highlight catchy findings
- We assume that positive results are automatically true
