Brainscape's Knowledge GenomeTM

Browse over 1 million classes created by top students, professors, publishers, and experts.


See full index

Scientific and Statistical Reasoning 2 > Lecture 11 Critical thinking about psychological research > Flashcards

Lecture 11 Critical thinking about psychological research Flashcards

1
Q

twee voorbeelden van “f you leave out enough, everything can become a beautiful story”

A
  • Publication bias, file drawer problem
  • Unspecified predictions, hidden nsustainable assumptions in theory
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
2
Q

Half of publications in 8 major psychology
journals (1985-2013) that involve NHST contain at least 1 error in reported p-values.

A
  • 1 in 8 contained a “grossly inconsistent p-value that may have affected the statistical conclusion”.
  • Those gross inconsistencies were more likely when significant vs non-significant results were reported
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
3
Q

researchers degrees of freedom

A
  • Which research area, which theory, which
    hypotheses?
  • How many dependent variables, how many
    conditions?
  • What measurement procedure?
  • How many participants?
  • What analyses? What outliers?
  • What is an effect? What is a relevant effect?
  • What can you conclude from the analyses?
  • What can you conclude from the investigations?
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
4
Q

issues with robustness =

A

when researchers that use the same data, find different effects.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
5
Q

only 49% of the papers published could be reproduced

A

oke

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
6
Q

waardoor komen die researcher degrees of freedom

A

door differences in robustness.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
7
Q

large researchers degree of freedom betekent

A

hogere variation
lagere robustness
lagere reliability

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
8
Q

hoe gaat het ideale process van wetenschap

A

author writes research -> editor decides who should review -> editor sends to experts in area -> experts review etc

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
9
Q

maar wat zien we helaas bij peer reviewers

A

mensen beoordelen significante resultaten als beter uitgevoerd, dan geen significante resultaten (hier is veel meer kritiek op)

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
10
Q

Goals of science do not always go hand
in hand with the goals of the scientist

A
  • Quantity of (frequently quoted) publications essential for further careers
  • The problem is that the pursuit of truthful/robust and interpretable results in the current system does not always lead to publications
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
11
Q

file drawer effect =

A

in science many results remain unpublished, especially negative ones.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
12
Q

Too much emphasis on new, surprising, findings is problematic:

A
  • We think too quickly that a published positive result is a full-fledged phenomenon.
  • Counterintuitive findings are looked at more, hoewel zij een lagere prior probability hebben
  • Small and noisy samples
  • Popular media will highlight catchy findings
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
13
Q

als de prior probability laag is, is rejecting the H0…

A

minder informatief

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
14
Q

kleiner sample =

A

kleinere power = minder informatief om H0 te rejecten

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
15
Q

type 1 error rate staat gelijk aan

A

alpha

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
16
Q

power is hetzelfde als

A

1 - B

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
17
Q

wat is B

A

type 2 error/false negative

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
18
Q

wat is een correcte true negative

A

1-a

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
19
Q

dus alles op een rijtje: formules in het vierkant van links boven naar rechtsonder

A

type 1 error/a, power/1-B, correct true negative/1-a, type 2 error/1-B

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
20
Q

hoe komt publication and reporting bias tot stand

A
  1. study publication bias
  2. outcome reporting bias
  3. spin
  4. citation bias
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
21
Q

study publication bias =

A

trials zonder resultaten worden niet gepubliceerd

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
22
Q

outcome reporting bias =

A

zelfs als de studie wel wordt gepubliceerd, worden de dingen zonder resultaat niet gepubliceerd in de final paper

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
23
Q

spin=

A

researchers beschrijven het alsof iets heel significant en met veel effect is

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
24
Q

citation bias =

A

dingen die werken hebben een hogere kans om geciteerd te wordne.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
25
Q

uiteindelijk kom je door publication and reporting bias dus bij een ding wat van 50/50% kans op werken is gegaan naar een geweldig resultaat wat sowieso werkt en veel consensus achter zit

A

oke

26
Q

wat was de bevinding van de dead fish brain activity

A

p-hacking

27
Q

p-hacking=

A

misuse of data analysis to find patterns in data that can be presented as statistically significant, thus dramatically increasing and understating the risk of false positives.

28
Q

alpha interpretatie van 0.05

A

if the null were true, we would mistakenly reject the null in 5% of the cases. we dont know whether we are right, but we know that we are not wrong most of the time.

29
Q

wat gebeurt er als je meerdere comparisons maakt

A

bv rollen met meerdere dobbelstenen en alleen de ene keer rapporteren dat je 6 gooide.

a type 1 error only applies to a single comparison that you are making: if you are doing multiple comparisons, the type one error inflates.

30
Q

wanneer gaat de probability of type 1 error omhoog (3 situaties)

A
  1. when measuring multiple dependent variables
  2. when comparing multiple groups
  3. when just testing out different covariates
31
Q

Type I error control would only work if …

A

we set up (and communicate) a clear sampling plan:
* Adding observations and testing after each new additions increases the probability of a type-I error
* So continuing data collection until an significant difference is found guarantees a type-I error (given that the null is true)

32
Q

in welke situaties gaat de p value steeds meer omlaag

A
  • meerdere dependent variables
  • meer observations per cell
  • dropping conditions
  • controlling for gender or gender interaction with treatment
  • alles gecombineerd
33
Q

why would power be lower in neuroscience papers

A

omdat zij hele dure apparatuur gebruiken

34
Q

de kans op het maken van een type 1 error inflates als je van een grote studie naar multiple small studies gaat

A

oke

35
Q

waarom is dit problematisch: if the p value is lower than 0.61 and if power is at least 0.35 then i should reject H0

A

omdat we dan niet meer in de meeste gevallen goed zitten (definitie van alpha)

36
Q

hogere sample = hogere power

A

oke

37
Q

wat is belangrijk bij replications van studies

A

je moet niet alleen checken voor significance! want als er een lack of sensitivity is, door een underpowered study, kan je nog een inconclusive replication attempt krijgen. dit betekent dat je nog steeds niet zeker weet welk resultaat je kan geloven. is the found effect in the replication meaningfully different from the original

38
Q

waarom hadden onderzoekers een effect gevonden van jonger kan worden door oudere muziek luisteren

A
  • hebben 1 conditie gedropped
  • optional stopping met datacollection
  • multiple dependent variables, only reported a couple
  • effect was dependent on covariate (but not mentioned)
39
Q

how do we solve these questionable research questions:

A
  • quality of research should be determining factor in how research/career is evaluated (not statistical significance)
  • replication should be more central (direct + conceptual replication)
  • importance of open science and pre-registration (online availability of data, materials, procedures and pre-publications.
40
Q

verschil direct & conceptual replication

A

direct = same research, different sample
conceptual = different research and sample, same concepts

41
Q

4 cognitive fallacies in research

A
  1. hypothesis myopia
  2. texas sharpshooter
  3. asymmetric attention
  4. just-so storytelling
42
Q

texas sharpshooter=

A

seizing on random patterns in the data and mistaking them for interesting findings. eerst data verzamelen en daarna pas een pattern zoeken tussen de bevindingen (gewoon schieten op een bord en daarna de cirkel er in tekenen) -> no way you could have predicted this in advance!

42
Q

hypothesis myopia=

A

collecting evidence to support a hypothesis, not looking for evidence against it, and ignoring other explanations

42
Q

asymmetric attention =

A

rigorously checking unexpected results, but giving expected ones a free pass.

42
Q

just-so storytelling

A

finding stories after the fact to rationalize whatever the results turned out to be

43
Q

waar komt hypothesis myopia mee overeen

A

confirmation bias

43
Q

waar komt asymmetric attention mee overeen

A

motivated reasoning: people tend to process information in such a way that it matches the goals they want to achieve. it makes ppl defensive (ook scientists)
-> looking more critically at unexpected results/think we already believe
-> looking less critically at predicted results

44
Q

wat is een voorbeeld van asymmetric attention

A

vrouwen keken meer sceptisch naar onderzoek dat aantoonde dat koffie leidt tot borstkanker, omdat zij dit zelf vaker krijgen (= attack op self -> self-defense -> meer critical)

45
Q

wat is dit voorbeeld van koffie

A

motivated skepticism (i want to be right)

46
Q

wat is de rule: 2,4,8

en wat kan hier mis gaan?

A

mensen denken er over na, schrijven dan op: bijvoorbeeld elke keer x2.

maar kan ook simpelweg zijn dat de nummers steeds hoger worden. dus er zijn meerdere explanations, je kan niet gelijk na het vinden van een uitleg uitgaan van die uitleg

47
Q

wat is de 2,4,8 rule een voorbeeld van

A

hypothesis myopia (alleen kijken naar de hypothesis waar je in gelooft, biased)

48
Q

if those who have been exposed to clutter (messy room) or a smell (fart spray) favour harsher punishments -> digust hypothesis

waar is dit een voorbeeld van?

A

hypothesis myopia

49
Q

wat is een black box model

A

what we know:

independent variable -> ? -> dependent variable

? = black box. dit kan meerdere uitleggen hebben.

50
Q

wanneer kan een false dilemma opkomen

A

als mensen bijvoorbeeld wel meerdere hypothesen geven over waarom ze data hebben gevonden, maar dat die ook nergens op slaan.

51
Q

wat is het verschil tussen texas sharpshooter en just-so storytelling

A

texas sharpshooter = patterns, story telling komt daarna. hangt dus heel erg samen.

52
Q

met welke vorm van bias is texas sharpshooter en just so story telling ook gerelateerd

A

hindsight bias (jezelf overtuigen dat je het wel van te voren had kunnen weten)

53
Q

3 principles tegen making errors=

A
  1. you must not fool yourself, you are the easiest to fool
  2. bend over backwards to try to prove yourself wrong!! be critical!!
  3. the essence of science is making mistakes in public: research papers should open up to allow others to help with corrections
54
Q

hoe heet het dat wij biased zijn voor onze eigen bias

A

bias blind spot

55
Q

hoe kan je deze bias blind spot tegen gaan

A

gebruik van actively open-minded thinking

56
Q

actively open-minded thinking (AOT)

A

1) search phase: search for possibilities (provide different possibilities and hypotheses that could also explain your data).
2) inference: if this were true, than this evidence is a lot more likely than if this were true.
3) goals and criteria
4) sufficiency, fairness and confidence

57
Q
A

Scientific and Statistical Reasoning 2 (11 decks)

Brainscape helps you reach your goals faster, through stronger study habits.
© 2024 Bold Learning Solutions. Terms and Conditions