errors and error detections

Question 1

what is the myth of self-correcting science

Answer 1

• humans are error-prone
• scientific process is not a given
• self correction does not happen by default - must take action

Question 2

what are the reactions due to errors

Answer 2

• concerns about data/results/conclusions
• error in data/analysis
• error in image
• error in method/text

Question 3

what are the consequences of self-retraction due to errors

Answer 3

• no damage to reputation
• positive effects of demo of values and integrity
• retractions due to misconduct not honest error meant drop in citations

Question 4

what are the types of errors

Answer 4

• careless errors (inattention, oversight, copy/paste errors, error in code)
• main effect disappeared so invalidating study’s initial conclusion
• negligent errors (incompetence, poor training, low quality design and analysis)

Question 5

what is the technique error for detection

Answer 5

• low cost
• easy to run
• rely on summary data from papers
• use reported design choices
• doesn’t always provide insight into error cause

Question 6

what is statcheck

Answer 6

• spellcheck for stats
• many numbers in stats tests are dependent on each other
• major types of inconsistencies checked
• cant handle certain types of tests or values

Question 7

what is the prevalence of stats reporting errors using statcheck

Answer 7

• 50% with at least one inconsistency
• 13% had a big effect on the study conclusions
• revealed biases

Question 8

what is the GRIM test

Answer 8

• granularity related inconsistency of means
• checks if the average makes sense given how many numbers in the set
• of 71 papers half reported at least one impossible number

Question 9

what is the randomisation check

Answer 9

• are randomised trials actually randomised
• ensures no systematic differences between the groups at the baseline

Question 10

what is the blinding check

Answer 10

• collects data is unaware of hypothesis and/or condition assignment
• runs stats analysis is unaware of hypothesis and/or group assignment

Question 11

what are the questionable measurements practices

Answer 11

• measurement schmeasurement attitude
• items in questionnaire made on the fly, lacking construct, mixing/matching items across scales, lacks internal validity
• measurement practices in large scale replications

Question 12

what is low power

Answer 12

• can only detect large effects which are exaggerated, unusual or spurious
• follow up studies use small samples expecting to find an effect
• underpowered studies are prone to detection

Question 13

what is self correction

Answer 13

• corrections submitted to journals
• adding/subtracting an integer to the F value so they would look statistically significant

Question 14

what are the global solutions

Answer 14

• systems of checks and balances
• data sharing
• pre prints
• open peer review
• full audit procedures
• scientists reputations are based on getting it right not being right

Question 15

what are the good practices for data analyses process

Answer 15

• state the theoretical hypothesis
• asses psychometric properties of variables
• plot univariate distributions
• plot a graph to match the theoretical hypothesis
• study residuals
• interpret parameter estimates and effect sizes
• replicate on a new data set
• make decision on the basis of a decision criteria