Chapter 11: Psychology and open science

Question 1

Replication crisis

Answer 1

36% of studies were replicated, lower for social psychology than for cognitive psychology (Open Science Collaboration, 2015)

Fraud increased scepticism towards psychology

Question 2

Why did Ioannidis argue most medical research findings were likely to be false?

Answer 2

There was a belief that one study finding a significant effect brings conclusive evidence for the effect

Question 3

Statistical power

Answer 3

Correctly rejecting null hypothesis

Question 4

Problems with replication

Answer 4

• misunderstanding of statistics (taking p-values as proof of effect)
• replication often done by same researcher
• direct replications were not considered interesting
• failed replications were not convincing and not published
• direct replications left for informal communications and beginning students

Question 5

File drawer problem

Answer 5

Non-significant studies are less likely to be published

Question 6

Conceptual replications

Answer 6

Do not directly replicate study

If no effect is found, it does not disprove, it introduces limiting conditions

Question 7

Questionable research practices

Answer 7

P-hacking: manipulating data to find a statistically significant p-value

• dropping bad participant data after analysis
• adding participants until statistically significant
• testing multiple dependent variables, only reporting significant ones
• adding covariates
• adding variables
• harking

Question 8

HARKing

Answer 8

Hypothesis after results known: creating a hypothesis that lines up with an effect found after analysing without research focus

Question 9

Optimistic views towards replication crisis

Answer 9

Researchers are now aware of their questionable research practices and the consequences, so will change behaviours

Question 10

Registered report

Answer 10

Evaluated registration of a study before it is conducted

Question 11

Why are Bayesian statistics better

Answer 11

They involve relative likelihoods and are more intuitive than traditional hypothesis testing that leads to misunderstandings

Question 12

Pessimistic views towards replication crisis

Answer 12

These problems have been known for a while and researchers will not change behaviours as they are too busy and do not have resources to do so

Question 13

Open science

Answer 13

Relevant information is easily available online to the public and other researchers, increasing transparency

Question 14

Repository

Answer 14

Location where data and analysis are stored, also included preregistration of studies with times

Question 15

TOP Transparency and Openness Promotion guidelines

Answer 15

Criteria created to describe how much journals adhere to open science practices with levels 0 to 3

Question 16

Standards for TOP guidelines

Answer 16

• citation of datasets and inspiration
• data transparency
• code transparency
• research material transparency
• design and analysis transparency
• preregistration of study
• preregistration of analysis
• replication

Question 17

Secondary data analysis

Answer 17

Using existing data to analyse different research questions

Question 18

Big data

Answer 18

Collection of large datasets used for secondary data analysis

Question 19

Characteristics of big data

Answer 19

Velocity of change
Volume
Veracity
Value
Variety

Question 20

Publish or perish

Answer 20

Researchers had to have a strong portfolio of research otherwise they would not be promoted; led to an increase in the number of studies published

Question 21

Increasing the quality of submissions in journals

Answer 21

• peer review
• journal impact factor based on number of citations

Question 22

Consequences of journal impact factor

Answer 22

• increased power of Garfield’s company and index
• dominance of research in English
• decreased importance of books compared to articles
• publication of articles with new findings to increase citations

Question 23

DORA Declaration on Research Assessment

Answer 23

• other sources than journals are important
• journal impact factors to be eliminated for promotions and funding
• research to be evaluated by itself and not based on journal
• explicitly saying which criteria used to make decisions

Question 24

Open access journal

Answer 24

Journal that can be accessed with no subscription or fees; usually charge APC article processing charges

EU decided all EU-funded research should be open access by 2020

Question 25

Double dipping

Answer 25

Traditional journals started open access journals, giving them two sources of income

Question 26

Predatory journal

Answer 26

Scientific journal that pretends to be genuine but is not so that researchers pay APCs

McKay and Coltheart (2017) sent nonsensical articles to journals and most accepted either with minor changes or once APC paid