Week 18 - Open Science

Question 1

What is effect size?

Answer 1

The measurement of the magnitude of an effect
-> effect of the difference between two groups

• Might not matter whether effect is large or small. Both can be interesting (small effect spread over millions of people can have large impacts)
• But if an effect is small, you need to make sure you have designed an experiment well enough to stand a chance of showing it
• Publication bias (only publishing results that have significant effects) may inflate th sizes of reported effect sizes

Effect size = Mean of condition1 - Mean of condition2 / Pooled SD
-> not affected by number of ppts
-> effect sizes are separate from p-values

Question 2

What is power?

Answer 2

Refers to the likelihood os getting a statistically significant result in this study given the sample size + the effect size
- Typically in behavioural sciences we must a power of 80%
- Replication studies should have the appropriate power to find the effect
-> issues with replication are often because the initial studies are often under powered
-> combine with publication bias towards exciting findings, this means thing get published which might not be as reliable as they claim

Question 3

What is open science approach?

Answer 3

• Movement to make parts of the research openly available to all for scrutiny
• Being open should reduce some of the systematic issues that have led to the replication crisis

Question 4

Why do we share?

Answer 4

Sharing data allows:
- Checking of analysis
- Consideration of how good the dataset is
- Comparison/combination with other datasets

Sharing the analysis:
- This will be a script like an R script
- Given the data and analysis script, people can replicate your findings exactly

Question 5

How do we share?

Answer 5

• Write a study protocol -> detailed written specification of:
  -Hypotheses
  -Methods
  -Analyses
  -> ideally level of detail would allow replication without instruction

Materials:
- Could be stimuli (words, pictures, videos), forms used, questionnaires used

Data + metadata:
- Raw data
- Anonymised data
- Anonymised data that is submitted to statistical analysis
- Need to be in sharable format
- Script for creating processed data from raw data made available if possible
- Metadata -> the documentation that explains the dataset e.g. who collected and how, how many variables

Analysis procedure:
- Exact specification of how moved from raw data to the results of the statistical analysis
- Includes how data was cleaned
- e.g. share R script

Question 6

Why might academics not like sharing?

Answer 6

• some of this is being cautious (sharing results that have taken years) and sometimes because it take time and planning up front
• Some are worried about being scooped -> other researchers scooping your idea and publishing it somewhere else
• Worried about ‘errors’ being discovered