Lecture 14: Critical thinking about statistical inference

Question 1

mindless statistic gaat over…

Answer 1

people are convinced that p values are the final say, if something is not p <0.05, it is not worth anything????

Question 2

statistics = probability -> nothing is certain!!

Answer 2

oke

Question 3

(1) You have absolutely disproved the null hypothesis (i.e., there is no difference between the population means).
(2) You have found the probability of the null hypothesis being true.
(3) You have absolutely proved your experimental hypothesis (that there is a difference between the population means).
(4) You can deduce the probability of the experimental hypothesis being true.
(5) You know, if you decide to reject the null hypothesis, the probability that you are making the wrong decision.
(6) You have a reliable experimental finding in the sense that if, hypothetically,the experiment were repeated a great number of times, you would obtain a significant result on 99% of occasions.

wat is er mis met ii, iv, v, en vi?

Answer 3

Options (ii) and (iv) may have been initially tempting, but hopefully you see now that as they refer to the probability of hypotheses they cannot be correct (an objective probability refers to a collective of events, not the truth of a hypothesis).

Option (v) is a sneaky one and often catches people out; but notice it refers to the probability of a single decision being correct; thus, option (v) cannot be correct (an objective probability does not refer to a single event) .

But option (vi) is a description of power, not significance.

Question 4

wat waren de mahony resultaten

Answer 4

mensen gaven prijzingen bij p<0.05, en waren meer kritisch bij p>0.05

Question 5

waardoor komen de mahony resultaten misschien

Answer 5

door misinterpretatie van de p value

Question 6

why do we use NHST at all?

Answer 6

weneed to understand whether the data-gathering process is:
- just randomness? (=sampling error and measurement error)
- just systematic variation?
- or both?

Question 7

welke rol neem je aan bij NHST

Answer 7

die van de devils advocate: if you think there is an effect, lets assume that there is no difference. is the random variation expected if there is nothing going on?

Question 8

wat was het process van significance testing according to Fischer

Answer 8

1. formulate H0: the hypothesis is to be nullified (dus niet perse 0, maar wij hebben dat er wel van gemaakt!).
2. report the exact level of significance -> p value. without further discussion about accepting or rejecting hypotheses.
3. only do this process if you know almost nothing about the subject. en het had een serie moeten zijn aan experimenten: als het repeatedly fails to give this level of significance.

Question 9

hypothesis testing according to neyman and pearson

Answer 9

1. formulate two statistical hypotheses, determine alpha, beta and N for the experiment
2. if the data falls in the H1 rejection region, assume H2. this does not mean that you believe H2, only behave as if H2 is true.
3. only use this procedure if there is a clear disjunction and if a cost-benefit assessment is possible.

Question 10

dus verschil fisscher en neyman & pearson

Answer 10

fisscher zei niks over hypothesis testing, alleen het rapporteren van de p value.

Question 11

lower alpha means lower power

Answer 11

oke

Question 12

wat is de relatie tussen alpha en beta

Answer 12

inverse relationship between alpha and beta—lowering one increases the other.

Question 13

wat gebeurt er als je alpha lager maakt

Answer 13

If you lower your alpha from 5% to 1%, you are simultaneously increasing the probability of making a Type II error, assuming all else is equal. Increasing the probability of a Type II error reduces the power of your test.

dus a en b zijn negatief gecorreleerd. power = 1-B en zal dus ook omlaag gaan als de alpha omlaag gaat.

Question 14

the null ritual =

Answer 14

(niet wat zij wilden!!)

1. set up a H0 -> no mean difference/no correlation. dont specify the predictions of your research etc.
2. use 5% as a convention for rejecting the H0
3. always perform this procedure

Question 15

wat zijn 3 denkfouten aan de null ritual

Answer 15

1. straw target: arguing against something that someone actually did not say (= zero correlation ding van Fisher)
2. common practice: 5% en always perform this procedure
3. false dilemma: if significant, always accept your research hypothesis

Question 16

we have reduced our scientific thinking to just checking significance

Answer 16

oke

Question 17

subjective probability=

Answer 17

probability is the degree of belief that something is the case in the world. (=uncertainty!)

Question 18

objective probability=

Answer 18

probability is the extent to which something is the case in the world (= relative frequency in the long term)

Question 19

wat zijn twee karakteristieken over p value die te maken hebben met probabilities

Answer 19

• gaat over objective/frequentist probabilities
• relates to conditional probabilities

Question 20

waarom kunnen we het niet hebben over “the probability of the null hypothesis”

Answer 20

omdat de p value een conditional probability is!!!

p(at least this test statistic | H0)

Question 21

dus als je …. ziet staan over de p value is het soweiso fout

Answer 21

iets met een statement over de nulhypothese! we weten alleen iets over de test statistic GIVEN the H0. dus die kan je niet zomaar omkeren!!!

Question 22

wat gebeurt er dus qua argumentvorm als je iets zegt over de H0

Answer 22

P -> B
B
P

dit is inversing the conditional probability + base rate fallacy (we ignore the base rate)

Question 23

hoe kunnen we deze probability wel inversen

Answer 23

we hebben daar de base rate voor nodig.

we would need a base rate on the null hypothesis: a hypothesis is true or false, there cannot be assigned a probability to this. there is nothing there!!!

Question 24

wat is hier mis mee: alpha equals the probability of making an error

Answer 24

because we have two errors: type 1 and type 2 error

Question 25

alpha is conditioned on…

Answer 25

the null being true.

a= P(what is the probability of making an error | H0 is true)

Question 26

wat gaat hier mis;

failing to reject the H0 is evidence for the H0

Answer 26

you should have a very very high power to claim this. net als met sleutels zoeken: heb je wel goed gezocht?

Question 27

failing to reject H0 is evidence for H0: 3 dingen die misgaan bij deze redenatie

Answer 27

1. je kan niet spreken van evidence, we hebben daar 2 hypothesen voor nodig.
2. je moet een power calculation doen om te kijken of je dit wel kan zeggen
3. power is de functie van: population size effect, alpha level en sample size. hangt dus van veel dingen af.

Question 28

power is the .. side of the distribution under the alternative hypothesis

Answer 28

right side

Question 29

als je een studie repliceert, waar moet je dan echt op letten

Answer 29

dat je niet hetzelfde aantal participanten gebruikt, want dan kan je studie underpowered zijn. dit komt namelijk doordat de power dan de rechterkant van de distributie is (=0.50 -> geen hoge power).

Question 30

the winners curse=

Answer 30

if you win an auction, you always pay to much (compared to the true value and to what other people are willing to pay.

Question 31

hoe kan je de winners curse zien in publication bias

Answer 31

maybe a lot of ppl are doing effect size estimates, but a lot are unsignificant. the ones that do find a significant value, get published -> overestimation of the effect.

Question 32

power is irrelevant when the results are significant: wat is hier fout aan

Answer 32

• P1 P<.05
• C I have found an effect
• (P2 When I have found an effect, it is no longer relevant what the probability is that I find an effect if H0 is not true)
• C2 Power is not relevant

hier zeg je iets over de alternatieve hypothese: je impliceert dat deze dus sowieso waar is. dat kan je niet zeggen, wij weten alleen iets over de H0 en dat die gereject kan worden.
you did not find an effect, you just rejected the null.

Question 33

QRP’s=

Answer 33

questionable research practices

Question 34

the garden of forking paths=

Answer 34

it is difficult to get a sense of how things would have turned out if something else changed, because of randomness

Question 35

neyman and pearson depends on…

Answer 35

the stopping rule you apply. this is difficult, because it is subjective.

Question 36

bayesian statistics praten over…

Answer 36

subjective probabilities, the degree of belief! je start hiermee met een prior belief en na data vinden -> allocate the credibility

Question 37

waarom kan je bij normale conditionele probabilities niet zeggen P (A|B) = P(B|A) ?

Answer 37

omdat we geen base rate hebben. dat kunnen we dus niet zeggen

Question 38

waarom kan je bij bayes wel dit berekenen P (A|B) => P(B|A) ?

Answer 38

omdat we hier wel een base rate hebben, die geven we er zelf namelijk aan.

Question 39

formule bayes

Answer 39

P(θ|D) = P(D|θ) * P(θ) / P(D)

Question 40

P(θ)=

Answer 40

prior beliefs

Question 41

P(D|θ)=

Answer 41

likelihood, the probability of the data given that a specific hypothesis is true

Question 42

dus wat vraag je je eigenlijk af bij de likelihood

Answer 42

hoe predictive zijn mijn verschillende hypothesen?

Question 43

P(D)=

Answer 43

how likely are the found data, given all the probabilities

Question 44

bayesian statistics is reallocation of … across …

Answer 44

credibility across possibilities

Question 45

credibility=

Answer 45

subjective probability or degree of belief

Question 46

possibilities=

Answer 46

how likely are different parameter θ values, given the data and our prior beliefs?

Question 47

hoe bereken je P(D|θ)

Answer 47

θ^D

dus bijvoorbeeld:
θ=0.1
D= 5 van de 5
P(D|θ) = 0.1^5

Question 48

dus wat is de likelihood

Answer 48

P(D|θ)

= the probability of the data given the hypothesis

Question 49

wat geeft de likelihood je

Answer 49

de distribution of the likelihood over the full range of the possible parameter values

Question 50

bayes factor formule =

Answer 50

P(D|θ1) / P(D|θ2)

Question 51

hoe heet de bayes factor ook wel

Answer 51

likelihood ratio, how much more probable is the data given one comparison, compared to the other? welke hypothese is more likely?

Question 52

posterior interpretation

Answer 52

degree to which you belief the hypothesis to be true, given the data

Question 53

wat gebruik je voor prior als je voor het experiment nog geen idee hebt

Answer 53

een flat prior (bij exploratory research)

Question 54

wat is een flat prior

Answer 54

each possible prior van θ is equally likely

Question 55

hoe narrower the peak van de prior…

Answer 55

hoe meer certain over θ

Question 56

wat is de relatie tussen P(θ|D) en P(D|θ)*P(θ)

Answer 56

dit is proportioneel!!!
als P(θ|D) 2 keer omhoog gaat, gaat P(D|θ)*P(θ) ook 2 keer omhoog!!!

dus posterior is proportioneel aan likelihood * prior

Question 57

P(D)=

Answer 57

marginal likelihood, hoe likely is deze data, vergeleken met de data voor alle andere mogelijkheden?

Question 58

wat is lastig aan bayes

Answer 58

• subjectief
• no error rate control

Question 59

(v) You know that if you decided to reject the null hypothesis, the probability that you are making the wrong
decision.

wat is hier mis mee?

Answer 59

Option (v) is a sneaky one and often catches people out; but notice it refers to the probability of a single decision being correct; thus, option (v) cannot be correct (an objective probability does not refer to a single event) .