problem 7 - psychological research

Question 1

what is the confirmation bias?

Answer 1

people tend to seek confirmation rather than disconfirmation of their beliefs

operates in at least 3 ways:
1. ambiguous info is interpreted to be consistent with one’s prior beliefs
2. tend to search for info that confirms rather than disconfirms their preferred hypothesis
3. more easily remember info that supports their position

Question 2

how do biases affect science?

Answer 2

without special protective measures biases (e.g. confirmation & hindsight) affect the scientific process - e.g. researchers seek to confirm, not falsify their main hypothesis

academic survival depends on how many papers one publishes = researchers find methods and procedures that maximize the probability of publication

primary concern: many published results may simply be false, as they have been obtained partly by dubious or inappropriate methods

Question 3

what is the fairy tale factor?

Answer 3

not drawing up an analysis plan in advance for viewing the data - consequence is that researchers can adjust their analyses to the data found = the results appear more convincing than they actually are

• increases the likelihood of presenting fictitious findings that are not replicable

Question 4

what is cherry picking?

Answer 4

researchers can measure many variables (gender, personality, age) and only report those that yield the desired result + can include in their papers only those experiments that produced the desired outcome

Question 5

what is massaging of data?

Answer 5

adjusting the data so that a significant difference emerges (such as post-hoc hypotheses)

Question 6

exploratory vs confirmatory research

Answer 6

exploratory: actively looking for a relationship where statistics are shaky
- the hypothesis is extracted from the data
- you will explore more

confirmatory: a healthy form of statistics - the analysis is determined in advance and then tested with the data
- the statistics leave no room for description

Question 7

causes of bad science: using the dataset several times

Answer 7

once you have used your data to formulate a hypothesis, this dataset is no longer useful for testing this hypothesis because a hypothesis developed from this dataset is unlikely to be refuted by the same data

so: if you hypothesize the data that inspired the test, your statistics are unreliable = double- dipping strategies

Question 8

pre registration of research

Answer 8

before you start the study, you publish the ‘protocol’ for it
- means that everyone can see what you’re going to do & measure, how you’ll do it, on how many ppl, etc before you start

fairy tale factor, confirmation bias and hindsight bias are eliminated

Question 9

how should research be conducted (in terms of exploratory & confirmatory)

Answer 9

can conduct exploratory research in the first phase of the research program → but these studies cannot be presented as strong evidence for any particular claim

in the second stage: purely confirmatory approach is desired - can be used as evidence for the hypothesis
- pre registration should be applied in this stage

Question 10

how can science be improved?

Answer 10

emphasizing replicability

emphasizing preregistration

making a clear distinction between exploratory and confirmatory research

Question 11

what is direct replication?

Answer 11

the attempt to recreate the conditions believed sufficient for obtaining a previously observed finding + is the means of establishing reproducibility of a finding with new data

may not obtain the original result

Question 12

what is HARKing?

Answer 12

hypothesizing after results are known

choosing the hypothesis which is most consistent with known results & most particularly with the results of the present research paper

whether the hypothesis was actually anticipated or seen as plausible prior to the execution of the study is of no concern

Question 13

reasons or incentives for HARKing

Answer 13

1. presumption that any worthwhile study always advances explicit, testable hypotheses
2. scientists tend to find confirmations more informative & valuable than disconfirmations (despite Popper’s arguments of the opposite)
3. research that advances one or more hypotheses & presents evidence confirming those hypotheses is more likely to be evaluated favorably for publication

Question 14

costs of HARKing

Answer 14

1. can translate into type I error
2. not communicating valuable info about what did not work
3. proposing theories that cannot pass popper’s disconfirmability test
4. disguising post hoc explanations as a priori explanations

Question 15

what are the publication and selective reporting biases?

Answer 15

publication bias = the differential choice to publish certain studies

selective reporting bias = reporting only particular results

Question 16

tests for publication & other reporting biases: tests of small study effects

Answer 16

evaluate studies included in the same meta-analysis & assess whether effect sizes are related to study size

when small studies have larger effects compared with large studies → may reflect publication or selective reporting biases (but alternative explanations exist)

Question 17

tests for publication & other reporting biases: selection models approaches

Answer 17

evaluate whether the pattern of results that have been accumulated from a number of studies suggests an underlying filtering process

Question 18

tests for publication & other reporting biases: excess significance tests

Answer 18

evaluate whether the number of statistically significant results in a corpus of studies is too high, under some plausible assumptions about the magnitude of the true effect size

Question 19

what is the replication crisis & how does this affect psychology?

Answer 19

there is a lack of replication studies
- credibility of the science is decreased
- without replications, it is hard to see if anyone committed QRP (questionable research practices)
- social science especially affected because it is hard to replicate those kinds of studies (subjective) - 25%
- cog psychology is a bit better at facing replication crisis - 50%

Question 20

what are potential reasons for the low replication rate in psychology?

Answer 20

• researchers want to find something new in the field = don’t do replication studies
• file drawer problem: gave up submitting a non-significant paper to the publisher
• time-lag bias: when sig results go through publishing first and then the insignificant ones get published later
• some studies are difficult to replicate due to: sample based on narrow demographic, complex methodology, or ethical boundaries