Lecture 11

Question 1

Methodolatry

Answer 1

Worship of the method of research over other facets of research such as interpretation.
Example of Pacific islands and american planes giving people supplies. Analogous to scientists following method but not understanding.

Question 2

Merton’s core values of science

Answer 2

• Communalism: scientific product belongs to no one
• Universalism: truth claims are judged the same, no matter who makes them
• Disinterestedness: scientists have no interest in the outcome of research
• Organized Skepticism: ideas are cracked down on and rigorously tested.

Question 3

Problems with communalism

Answer 3

Often scientists regard their data as their own and don’t share it or materials and analysis code.

Question 4

Problems with universalism

Answer 4

There are clearly power structures (editorial positions, leaders of scientific organizations, senior researcher in faculty).
Underrepresentation of women and minorities.

Question 5

Problems with disinterestedness

Answer 5

Researchers have an interest in succes because they want to be published. This causes more fraud and trying to play with the numbers.

Question 6

Problems with organized skepticism

Answer 6

Scientific culture seperates the idea and the person. But sometimes people do argue in a personal way.

Question 7

Standard research practice in the 20th century

Answer 7

• Create theory (verbal story)
• Come up with an experiment (usually with a small n)
• Test hypothesis (significance test)
• Publish results (no disclosure of data)

Question 8

Questionable research practices

Answer 8

• P-hacking: deleting outliers, adding covariates, splitting the sample, selecting variables until effect found
• Publication bias: only publishing a positive result
• HARKing: hypothesizing after results are known
• Dropping experimental conditions that don’t work.

Question 9

Crisis of confidence

Answer 9

In 2013 a group of researchers assessed replicability of 100 psychological studies. They found out many could not be replicated

Question 10

Pillars of open science

Answer 10

• Open Data: anyone can assess the evidence
• Open Materials: anyone can replicate
• Preregistration: no a posteriori tinkering with analyses

Question 11

Brian Nosek

Answer 11

Developer of Center for Open Science, GitHub etc.

Question 12

The Manylabs project

Answer 12

Project to replicate a lot of studies with ‘sexy findings’.
Found that many don’t replicate. Studies with smaller effect sizes often do replicate.

Question 13

Criticisms of Open Science

Answer 13

• More bureaucracy and more work for researchers
• Most data isn’t downloaded, many preregistrations aren’t checked
• Too much focus on replication, not enough on creativity
• Expensive
• Not always feasible
• Some approaches like field studies and qualitative research may decline