The Crisis & Open Science Flashcards
Define:
‘Researcher degrees of freedom’
(Construct courtesy of ‘Flase-Positive Psychology […]’ paper)
And what kind of research issue typically arises due to it?
The many ‘exploratory behaviours’ researchers can apply toaddress ambiguity during the collation and analysis of data (e.g. exclusion of some observations, termination/continuation of data collection, etc.)
As researchers often tend towards ‘self-serving’ adjustments (i.e. those that will likely yield a result that supports their initial hypothesis/research question), ‘researcher degrees of freedom’ often leads to Type I Errors, i.e. False Positives
Identify:
According to the ‘False-Positive Psychology’ article, what are the four main types of ‘researcher degrees of freedom?’
Decisions surrounding…
- Selection amongst dependent variables
- Sample size
- Covariate use
- Experimental condition subset(s) reporting
Note: researchers may also employ a variety of combinations of these different degrees of freedom, which can lead to a wide range of effects on their final results (i.e. provide lots of avenues leading to a ‘statistically significant’ finding)
Identify:
Give some of the major reasons for why False Positives may be so prevalent in published scientific research:
(Derived from ‘False-Positive Psychology’ paper)
- ‘Null results’ in replicative research are not conclusive (as many factors could lead to them and so cannot definitvely prove the occurrence of an F.P.)
- Reknowned journals/publishers rarely include research that have null conclusions or are exact replications, so there is low incentive to invest in these
(therefore lowering likelihood of checking for F.Ps) - Research containing them tend to ‘syphon’ more resources/time-investment (due to perceived novelty/innovative-nature)
- Publishers may be concerned revealing already published papers pertaining to false positives could lead to negative impacts on their reputation
Define:
False Positive
(i.e. Type I Error)
A statistical error whereby you ‘reject a null hypothesis that is actually true (for the population)’
(e.g. a pregnancy test yielding a positive indicator when the woman is not actually pregnant)
Define:
False Negative
(i.e. Type II Error)
A statistical error whereby you ‘fail to reject a null hypothesis that is actually false (for the population)’
(e.g. a test for a virus coming back ‘negative’ when the person does have that particular illness)
Define:
Covariate(s)
‘(Continuous) Variables expected to correlate with changes seen in the outcome/dependent variable
Often, covariates are used to help rule out alternative explanations in findings or help explain some of the variations seen in the dependent variable
(e.g. age is a common example of a covariate in many studies)
What is the ‘file-drawer’ problem in scientific research?
This refers to the fact that the majority of published research are only those that contain ‘significant findings’, while those that don’t tend to ‘never see the light of day’.
(i.e. they get ‘filed-away’ despite potentially containing valuable data and insights for certain theories).
List:
The SIX proposed ‘Requirements for Authors’ in research to minimise the ‘Problem of False-Positive Publications ‘
(According to the ‘False-Positive Psychology’ Article by Joseph P. Simmons; Leif D. Nelson; Uri Simonsohn)
- Determine the rule for terminating data collection beforehand and report this in the article.
- Collect at least 20 observations per cell (or provide ‘compelling justification’ otherwise).
- List all variables collected in the study.
- Report all experimental conditions (including failed manipulations).
- Report what the statistical results would have been if any removed observations were still included.
- Report the statistical results of the analysis if any included covariate(s) were to be removed.
List:
The FOUR proposed ‘Requirements for Reviewers’ in research to minimise the ‘Problem of False-Positive Publications ‘
(According to the ‘False-Positive Psychology’ Article by Joseph P. Simmons; Leif D. Nelson; Uri Simonsohn)
- Ensure that authors follow their requirements.
- Show more tolerance for imperfections in results.
- Require the authors to demonstrate their results do not hinge on any arbitrary analytic decisions.
- Ask authors to carry out replications if discrepencies are detected in their justifications for data collection and/or analysis.
What is the ‘Replication Crisis’ in the context of Psychology?
This refers to the aftermath of a study conducted in 2015 whereby a group of Psychological researchers set out to ‘estimate the reproducibility of Psychological science, and found that only ~40-50% of their replications produced similar results to the original studies.
What are the THREE main critiques for the ‘Replication Crisis’ study (2015)?
- ‘Researcher expectations’ may have influenced the findings/study.
- Quasi-random assignment of studies to researchers was used.
- Researchers for this study were self-selected.
‘Quasi-random’ in this case entailed allowing researchers to select which study they wanted to retest from a subgroup of randomly selected studies - meaning they could have potentially chosen ones they thought would be less likely to produce similar results when replicated.
What are QRPs?
(In research contexts)
‘Questionable Research Practices’
What THREE categories of QRPs lead to no replication of some studies?
(‘Questionable Research Practices’)
- Fraud.
- Mistakes.
- ‘P-Hacking’ & ‘Stop-Start Science’.
What THREE categories of QRPs lead to no continuity from some studies?
(‘Questionable Research Practices’)
‘No Continuity’ refers to a study’s inability to add to the scientific method ‘cycle’ for the systematic accumulation of knowledge.
- ‘HARK-ing’.
- ‘The File-Drawer Problem’.
- Not Sharing.
- ‘Hypothesising After Results are Known’ (i.e. ‘pseudo-deductive’). Instead the researcher should disclose their methods were exploratory/inductive (as they first recognised patterns in the data, etc.).
- This is where research with ‘non-significant’ findings tend to have less chance of being published. In other words, only ‘significant’ data/findings is permitted by such practices to contribute to the ‘Scientific Method Cycle’ (see image).
This can lead to an over-inflation of theories and their significance. - This violates ethical standards (especially in the case when the researchers themselves prevent others from accessing their work/findings). It can take other forms too though, such as putting data/studies behind ‘pay-walls’.
What is ONE common example of fraud in research?
Inventing data/numbers
(to skew results a certain way).
Define:
‘P-Hacking’
When scientists ‘misuse’ their researcher degrees of freedom to manipulate the study towards achieving significant results.
An example of this is ‘Stop-Start Science’ in which the researcher(s) periodically check the data during the collection process to assess whether a significant test-statistic has been found (and stopping there if it has, or continuing to collect data if not).
P-values are subject to fluctuation over time due to random chance, and so if you keep monitoring these changes you may eventually find a random ‘dip’ to a significant value.
However, it will eventually restabilise with the more data you collect to a non-significant value level (if there is in fact no correlation in the data).
