Trust Economics Week 9

Question 1

This topic explores how much we should trust empirical results published in journals.

Question 2

FACTS: replicability vs non-replicability
Replicability exists in sciences why we have law of physics etc.

Non-replicability in economics as studying animal behaviour (no heterogeneity when responding to stimuli)

Question 3

What is the standard economic method of testing

Answer 3

Hypothesis testing i.e start with null.

E.g drug doesn’t cure cancer. (No relationship on cancer)

Reject null means drug does cure cancer. (There is a relationship)

Question 4

P value

Answer 4

How unlikely the pattern in your data would have arisen, if null is true. (E.g drug does not cure cancer, but how surprised am I to see data that shows it does?) (level of confidence we can reject the null, smaller=better)

(Small p value=greater evidence null hypothesis is false, so drug does cure cancer)

Question 5

2 problems with research practices harming credibility/trustworthiness of results

Answer 5

Publication bias

P-hacking

Question 6

Publication bias

Answer 6

Publication likelihood greater with a smaller p-value
I.e smaller p-value = ones that have a significant effect, and reject the null.

Question 7

P-hacking

Answer 7

Researchers make methodological choices in conducting their studies that tend to deliver lower P-values i.e significant effects. (Again no-one wants boring new with no relationship)

Remember P-hacking can still be accurate, just bias.

Question 8

What is publication bias an example of?

Answer 8

A selection effect

Question 9

Selection effect and example

Answer 9

What we see in newspapers/journals is not everything, but a filtered subset. Selection is not usually random, so needs to be considered!

e.g the WW2 Study by RAF of returning bombers and their pattern of bullet holes. TEACHES YOU TO ACKNOWLEDGE HOW SELECTION HAPPENED, AND THE SUBSEQUENT REASON FOR RESULTS

Question 10

Why does publication bias arise

Answer 10

People want to see X is related Y, not X isn’t related to Y!

E.g people wanna see coke makes you bald, not coke doesn’t make you bald.

Question 11

Opposed to nature of science, how does nature of statistics work? And examples

Answer 11

Enough studies enough times can create a purely spurious result

Jelly bean colour example, coke example.

Unlike where you apply heat to water, it always becomes steam.

Question 12

Coke example explained

Answer 12

Questions men about their consumption for 15 drinks. Then whether bald or not.

Most drinks had nothing to do with hair loss, represented by blue dots.

Journalist only writes up on red dot (significant result)
No causation, but a positive correlation between coke o drinkers and being bald. (Maybe cos older men drinking more fizzy drinks!)

Question 13

Examples of p-hacking decisions the researcher can make (3)

Answer 13

What data to collect
What sample
Defining variables

Question 14

Example of p-hacking in hotel reviews

Answer 14

Hotel only request feedback for online review if they know their customers enjoyed their stay, hence why reviews are generally good on tripadvisor

Question 15

Results can be probabilistic rather than deterministic- example

Answer 15

E.g giving employee of month might not motivate every worker, but a change in effort across a large sample/portion

Question 16

We need to consider scale of these problems.

Question 17

Stylised example without publication bias or p-hacking

Answer 17

Consider 1000 hypotheses, 50 true. We don’t know which. So experiment done on all 1000.

Experiment finds significant result if actually true=0.8 (“Power” of test)
Experiment finds significant result if actually false=0.05 (arbitrarily p-value cut-off)
Seeing a positive result, the chance of underlying hypothesis being true is

(0.8 x 50) / {(0.8 x 50) + (0.05 x 950)}= 0.46

Question 18

Meaning of this

Answer 18

If we see a paper at a 5% significant that the result is true, the probabitly of actually being true is 0.46

LESS THAN HALF!

This example shows the extent of the problem, REMEMBER THIS EXPERIMENT HASN’T EVEN HAD PUBLICATION BIAS OR P-HACKING, SO IN THE REAL WORLD EFFECT EVEN WORSE

Question 19

So what should we consider more (2)

Answer 19

Theory along with results to support the findings

High powered studies, more compelling with a bigger sample.

Question 20

Marathon analogy

Answer 20

We would assume a small amount of people at the beginning, a lot in middle range and few at ending- a normal distribution.

In reality there are multiple spikes; people push to get 2:59 rather than 3:01 etc. so these cause the spikes with more people finishing at these times

Question 21

Brodeur et al

Answer 21

Collected 13440 p-values to see their z-statistics distribution when unmanipulated!

Spikes around critical values (like the marathon example).

Show a major filtration process, meaning we don’t see certain results! (Ones where fail to reject i.e the boring ones)

Question 22

So first way to collect evidence is to collect P-values and see the distribution (marathon and Brodeur).

Second way to see more evidence, and example

Answer 22

Try to replicate studies

If we collect a bunch of studies with p<0.05, this means more than 95% should replicate!

Nosek- replicated 100 research findings, only 39 could be reproduced. (39 IS NOT 95!!!)

Question 23

So, 2 ways to show evidence and increase accuracy

Answer 23

Collect p-values and see their distribution

Replicate studies

Question 24

Solutions

Answer 24

Change publication practices to allow null results to be more easily published.

Encourage replications via data sharing, journals publishing replications, university recognition etc.

Make replications easier e.g via relaxing data-sharing requirements: “open science” movement

Require “pre-registration” or “pre-analysis plans” to address p-hacking

Question 25

Change publication practices to allow null results to be more easily published. Example

Answer 25

Journal of Development Economics receive experiments without results. So reviewer and editor only decide to publish on the basis of the quality of the methodology, rather than the answer!

Question 26

Require “pre-registration” or “pre-analysis plans” to address p-hacking

Answer 26

Make experimenters state their choices and state what they’re going to do, before collecting their data.

So reviewers know how these results arised prior.