Brainscape's Knowledge GenomeTM

Browse over 1 million classes created by top students, professors, publishers, and experts.


See full index

SSR 2 > L2 - Critical thinking about psychological research > Flashcards

L2 - Critical thinking about psychological research Flashcards

1
Q

Replicability

A
  • same research but with different sample
  • procedures are done as equal as possible to original study, but with different participants
    = if different results, then study lacks replicability
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
2
Q

Reproducibility

A
  • extent to which others can reproduce the findings using same data and same data analysis protocol
  • no different sample and no different RQ
    > e.g. tried to reproduce economics studies, and more than half were not reproducible
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
3
Q

what are some possible reasons for lack of reproducibility?

A
  • data was not provided
  • analysis protocols were not provided
  • authors did not respond
  • …
    > we fail 50% of times sometimes as well for no apparent reason
    ! could explain also replicability
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
4
Q

Robustness

A
  • same data to different researchers, different analysis
  • they get different results based on analysis chosen
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
5
Q

Researcher degrees of freedom

A
  • all the individual decisions that the researchers have to make to analyse data (choices made are not same for everyone)
    > e.g. what measurement procedure? what analysis? how many participants? what is a relevant effect? …
  • if researcher is biased, it will largely impact results because of his degree of freedom
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
6
Q

what are the steps of the publication process?

A
  1. researcher writes manuscript
  2. sends it to the editors
  3. reviewers review it
  4. suggestions for revision and publication
  5. author fixes it
  6. study gets published
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
7
Q

what are the studies reviewed on before publication?

A
  • clarity
  • accuracy
  • appropriate methodology
  • theoretical base
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
8
Q

What is a case that explains biases in reviewing?

A
  • same exact paper but different direction of the results (in some cases expected results occurred, in others they didn’t)
  • different peer reviewers in different conditions led to very different reviews
    = “expected results condition”: basically no criticism, very positive feedback
    = “not-expected results condition”: harsh feedback, much criticism
    !! high influence of expected results on reviewer’s judgement
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
9
Q

what is the issue with not publishing studies with insignificant results?
(e.g. no effect found)

A
  • the researchers start pursuing only significant results, which lead to biases
    > might want to move their carreer further and not lose their jobs, which might happen if they don’t publish
    → goal of science and of scientist does not allign anymore
    > study showed that 96% of results in studies were the expected ones
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
10
Q

File drawer effect

A
  • not publishing studies with results that were not expected
  • it is estimated that 50% of studies in psychology remain unpublished
  • leads to publication bias
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
11
Q

what is usually overemphasized in studies?

A
  • counterintuitive findings
  • small and noisy samples
    ! too much emphasis on new, surprising findings is problematic
    (see picture 2)
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
12
Q

what are counterintuitive findings in terms of conditional probability?

A
  • prior probability: unconditional probability of hypothesis being true regardless of the research results
    → in counterintuitive findings, the prior probability is very low (= low base rate)
    = low prior probability of alternative hypothesis being true
    (to be rechecked)
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
13
Q

what is the problem when using small and noisy samples?

A
  • with smaller and noisier samples, power is smaller as well
    → increased type II error
    → more likely to not reject the null hypothesis when it is false
    = lower probability of null hypothesis being false given that it was rejected
    (to be rechecked)
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
14
Q

Publication and reporting bias

A

(see picture 1)
- from original studies, (eg) 50% found no effect
1. publication bias
2. outcome reporting bias
3. spin
4. citation bias
= at the end, almost all studies point at intended effect

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
15
Q

Publication bias

A
  • some studies with not expected results are not published
    > all studies with positive results and only some with negative results are published
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
16
Q

Outcome reporting bias

A
  • some variables and conditions are left out of published studies with initial negative results
  • some questionable practices are applied
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
17
Q

Spin

A
  • results are interpreted in vague and not accurate way, making results sound significant
    → doesn’t stand out as much that results weren’t as predicted
    > e.g. “(insignificant) effect is significantly more marked than placebo”
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
18
Q

Citation bias

A
  • positive results are cited more than studies with negative results
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
19
Q

p-hacking

A
  • being on the lookout for significant findings
  • data analysis carried out to so that findings are significant
    > one of the questionable research practices
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
20
Q

Type I & Type II error

A
  • Type I error: rejecting H0 when true (usually 5% of cases)
  • Type II error: not rejecting H0 when false (usually 20% of cases; power = 80%)
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
21
Q

what is the ideal scenario when deciding analyses?
what is common practice instead?

A
  • cost & benefit analysis of using certain power and alpha level
  • usually, people just use power of 80% and alpha of 5% (default value)
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
22
Q

What is the effect of questionable research practices on the interpretation of the results?

A
  • they inflate the likelihood of type I error
  • instead of using one outcome, they use more of them, leading to an increased change of getting expected outcome
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
23
Q

how does the probability of getting Type I error change based on how many tests we run?

A
  • The probability of making type I error in individual test is 0.05, but probability of type I error for whole collection of tests is higher
  • p-hacking
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
24
Q

in what occasions would the probability of getting type I error be higher?

A
  • when measuring multiple dependent variables
  • when comparing multiple groups
  • when making testing a difference with or without including a covariate
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
25
Q

how can we control for type I error? why?

A

By setting up and communicating clear sampling plan:
> adding observations and testing after each new addition increases the probability of Type I error
> so continuing data collection until a significant difference is found, guarantees T.1 error

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
26
Q

What are the estimation of actual power in the studies?

A
  • it is originally set as 80%
  • estimates range grom .50 to .35
    >.31 in specific domains (e.g. neuroscience)
    !! so power is quite low, but 96% of published studies have expected results → this indicates mismatch in percentages
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
27
Q

Why is “checking for significance” an issue?

A

Because of:
- lack of sensitivity
> underpowered studies make for inconclusive replication attempts
> 49% of replications were inconclusive (but are often reported as conclusive failures to replicate)
- lack of differentiation
> is the found effect in the replication meaningfully different from the original?

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
28
Q

what are some solutions for all the questionable research practices?

A
  • quality of research should be determining factor in how research is evaluated (and not statistical significance)
  • replication should be more central (both direct and conceptual)
  • open science and pre-registration (online availability of data, materials, procedure and pre-publication)
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
29
Q

what are direct and indirect replications?

A
  • Direct: same research, different sample
  • Conceptual: different research and sample, same concept
30
Q

Registered report

A
  • publisher guarantees to publish study no matter what results are
  • around 45% of predicted results are met (instead of 96%)
31
Q

Why are questionable practices so common in research?

A
  • most times they are not intended
  • there are various different outcomes that a study could have, based on the researcher’s degrees of freedom
  • sometimes, the outcome is the consequence of a series of decisions
32
Q

what other decisions could have been made by the authors of the study on fertility and religion?
- do not memorize, it is just to understand previous flashcard

A
  • different cycle days (still reasonable)
  • relationship status (ambiguous questions)
    → this leads to 180 different possible outcomes (if other decisions were taken)
    (see pictures 3 & on)
    ! hard to assess how robust the finding is, if researchers are not open about the decisions made
33
Q

Asymmetric attention

A
  • bias in critical thinking
  • rigorously checking unexpected results, but giving expected results a free pass
  • motivated skepticism (skepticism biased towards things we don’t want to accept / agree with)
33
Q

Black box argument

A
  • (hypothesis myopia)
  • something happens, and we observe the response
  • we then infer the interpretation of the process
    (see picture 6)
34
Q

hypothesis myopia

A
  • bias in critical thinking
  • collecting evidence to support a hypothesis, while not looking for evidence against it, and ignoring other explanations
  • prove hypothesis without looking for alternative interpretation
35
Q

Texas sharpshooter

A
  • bias in critical thinking
  • seizing on random patterns in the data and mistaking them for interesting findings
36
Q

Just-so storytelling

A
  • bias in critical thinking
  • finding stories after the fact to rationalize whatever the results turn out to be
  • researchers are not necessarily aware of how their decisions influenced results, so they think that results are accurate and representative of state of the world
37
Q

Actively Open-Minded Thinking (AOT)

A
  • we should take into consideration all possibilities, evidence and goals
  • pay attention to sufficiency, fairness and confidence
    > e.g. where all possibilities considered? was the evidence considered in light of all hypothesis? were the relevant criteria applied? …
38
Q

Article 2

What are the two main problems in research today?

A
  • scientific field is more competitive than ever > emphasis on piling up publications with statistically significant results
  • not good-enough tools when considering multiple variables
39
Q

Article 2

What is Hypothesis Myopia?

A
  • collect evidence to support just one hypothesis
  • not look for evidence against it
  • fail to consider other explanations
40
Q

Article 2

what is an example of hypothesis myopia?

A
  • Sally Clark convicted of murdering her two sons, because Sudden Infant Death Syndrome appeared very unlikely to happen twice in one family
  • failed to consider base rate of double murder happening in a family
  • likelihood ratio of 9:1 (SIDS:against murder)

= they failed to account for other hypothesis and explanations for an event, only collected explanation for their initial hypothesis

41
Q

Article 2

What is the Texas Sharpshooter?

A
  • “drawing the target around the pattern of bullets already shot”
  • pick the one option that explains the most agreeable results
42
Q

Article 2

what is p-hacking?

A
  • exploiting researcher degrees of freedom until p<0.05
  • misuse of data analysis to find patterns in data that can be represented as statistically significant, thus increasing the risk of false positives
  • e.g. perform many statistical tests and only report those that came back with significant results
43
Q

Article 2

What is HARKing?

A
  • report unexpected findings as having been predicted from the start
  • hypothesizing after the results are known
44
Q

Article 2

What is asymmetric attention?

A
  • giving expected results a free pass, but rigorously check non-intuitive results
  • e.g. 88% of cases in which results did not allign with hypothesis, the inconstistencies were blamed on how experiments were conducted, not on the theory of the researchers
45
Q

Article 2

what is Just-so Storytelling?

A
  • justifying results that come up after obtaining them
46
Q

Article 2

what is JARKing?

A
  • justifying after results are known
  • rationalize why results should have come up a certain way but did not
47
Q

Article 2

what are some solutions to the bias in researching?

A
  • strong inference:
    > explicitly considering competing hypotheses + develop experiments to test for them
    > tackles hypothsis myopia
  • explicitly listing alternative explanations
    > reduce just-so storytelling
48
Q

Article 2

Transparency

A
  • share methods, data, computer code and results
  • register reports (presenting plan for peer review before they do experiment)
49
Q

Article 2

Team of rivals

A
  • adversarial collaboration (proponent-sceptic)
  • team up with “rivals” in the field to get to the truth
  • hard to carry out, because it’s hard for researchers to team up with people that will try to dismantle their research
50
Q

Article 2

Blind data analysis

A
  • researchers who do not know how close they are to desired results will be less likely to find what they are unconsciously looking for
51
Q

Article 2

how can blind data analysis be carried out?

A
  • write a program that creates alternative data sets (eg add random noise or move participants to different conditions)
    > they carry out analysis of fake results and only at the end they get real results, and analysis cannot be changed fiddled with
52
Q

Article 3

What are the problems with false positives?
(Rejecting the null hypothesis when true)

A
  • particularly persistent mistake in research
  • research have little incentive to find null results (they will not be published)
  • false positives waste resources
  • risk of losing credibility of scientific field of researches with published false positives
53
Q

Article 3

how do false positives come to be?

A
  • because of the reserachers’ degrees of freedom
  • with all the decisions that researchers can make in the study design and analysis plan, the likelihood of false positives is higher than 0.05 (alpha)
54
Q

Article 3

where does this exploratory behavior stem from?

A
  • ambiguity in what decision is best
  • researcher’s desire to find statistically significant result
55
Q

Article 3

what are some possible degrees of freedom of researchers (rDf)?

A
  • choosing among dependent variables
  • using covariates
  • reporting subsets of experimental conditions
  • choosing sample size

! experiment was conducted over these rDf, and it showed that they would increase the false positive rate up to 50%

56
Q

Article 3

What are the requirements for authors?
- solution to problem of false-positive publications

A
  1. atuhors must decide the rule for terminating data collection before it begins, and report rule in the article
  2. athors must collect at least 20 observations per cell or provide compelling cost-of-data-collection justification
  3. authors must report all experimental conditions, including failed manipulations
  4. if observations are eliminated, authors must also report what the statistical results are if those observations are included
  5. if analysis includes a covariate, authors must report the statistical results of the analysis without the covariate
57
Q

Article 3

Sample-size rDf

A
  • many researchers stop data collection on basis of interim data analysis (estimated 70%)
  • e.g. stop collecting data when statistical significance is obtained or when n of observation =50
  • based on wrong idea that effect significant with small sample size is also significant with larger sample size
58
Q

Article 3

what are the guidelines for reviewers?
- solution to problem of false-positive publications

A
  1. reviewers should ensure that authors follow the requirements
  2. reviewers should be more tolerant of imperfections in results
  3. reviewers should require authors to demonstrate that their results do not hinge on arbitrary analytic decisions
    > reviewer should ask for alternatives of the arbitrary decisions made by author
    > ensure that arbitrary decisions are consistent across studies
  4. if justifications of data collection or analysis are not compelling, reviewers should require the authors to conduct exact replication
59
Q

Article 3

what is wrong with the study about listening to music and age felt?

A

see image 7

60
Q

Article 3

file-drawer problem

A

only reporting the experiments that work

61
Q

Article 3

what is a solution for the file-drawer problem?

A
  • asking researchers to submit studies independently from result
    > how to enforce submission?
    > how to ensure disclosure of degrees of freedom?
  • publishers should give incentives and reinforce disclosure practices, until it becomes common
62
Q

Article 3

what are other possible solutions for rDf? what are their criticisms?

A
  • Correcting alpha levels
    > might be interpreted as ulterior rDf
    > no clear effect of specific rDf on findings, therefoe no clear direction of correction of alpha
  • Using Bayesian statistics
    > increases rDf (additional judgements of prior d.)
    > new set of analyses that could be subjected to data
  • Using Conceptual replications
    > might choose different conditions and report different measures
  • Porting materials and data
    > too big cost on reader and reviewer
    > allows for reduction of a condition from raw data as well (= no transparency)
63
Q

Article 1

what are two widespread problems of science?

A
  • science is ignored
    > judgements that could be better made by computers are made by humans (e.g. selection procedures in university)
  • “replication crisis”
    > most studies cannot be replicated
    > journals will publish studies with most surprising results (= with higher chances of being wrong)
64
Q

Article 1

Notes - bias in science

A
  • journals are becoming more concerned with financial conflict of interest (studies showing results that sponsors want)
  • cognitive dissonance is present when researchers defend their positions despite evidence of them being wrong
65
Q

Article 1

Actively open-minded thinking (AOT)

A
  • thinkers should be open to challenges and seek out challenges proactively
    > e.g. thinking of alternative possible conclusions, asking questions about interpretations of conclusion, …
66
Q

Article 1

statistical control

A
  • persistance in methodology used
    > often interaction effects are interpreted badly and interactions disappear when we change measurement tools
    > we must not interpret interactions unless the additional variable reverses the direction of the effect (cross-over interaction)
67
Q

Article 1

what are the bases of AOT?

A
  • search
    > possibilities (possible answers to questions we are asking)
    > evidence (facts and beliefs that influence the evaluation of possibilities)
    > goals (what we apply to evaluate possibilities in light of evidence)
  • inference
67
Q

Article 1

conjectures and refutations

A
  • don’t wait around for contrasting results, look proactively to more data as clearly as possible
68
Q

Article 1

My-side bias

A
  • tendency to search for reasons supporting a favoured conclusion
  • ignoring alternative explanations, …

SSR 2 (12 decks)

Brainscape helps you reach your goals faster, through stronger study habits.
© 2024 Bold Learning Solutions. Terms and Conditions