Misleading Data

Question 1

What is publication bias?

Answer 1

The fact that in many fields only papers with significant results are published, and papers without will not be published.

Bias towards significant effects

Question 2

How can you also call publication bias?

Answer 2

Filedrawer problem

Question 3

What are the 7 deadly sins of psychology?

Answer 3

1. Confirmation bias
2. Hidden flexibility | data tuning
3. Unreliability | replication crisis
4. Data hoarding | Failure to share data
5. Corruptibility
6. Internment
7. Bean counting | publication counting

Question 4

What is definition of confirmation bias?

Answer 4

The tendency that people have to embrace information that supports their beliefs and reject information that contradicts their believes

Question 5

In Peter Wason’s Selection Task which cards does the people turn over to test the truth of the rule?

Answer 5

the rule was: if a card has a vowel on one side, then it has an even number on the other side.

E K 2 3

Need to turn over E and 3.

Question 6

Which card in Peter Wanson’s selection task did we turn over often?

Answer 6

E K 2 3

we turned over 2 often

Question 7

What could be confirmed with that we turn over 2 often?

Answer 7

We tend to look for confirmation (turning the 2) instead of falsification (turning the 3).

With 3 we disconfirm the rule but that is something we rather not do.

Question 8

What is null hypothesis significance testing?

Answer 8

• H0 and H1
• P = the probability of an effect if H0 were true but NOT the probability of H- itself being true

if P > 0.05 –> H1 is rejected BUT H0 cannot be accepted.

• inherent bias towards confirming an effect. You cannot confirm that there is no effect.

Question 9

What are the degrees of freedom in psychological research according to Jelte Wichers?

Answer 9

1. Observe (literature)
2. Formulate hypothesis
3. Prediction (design)
4. Test (Data collection & analysis)
5. evaluate (present)

Question 10

What is exploratory research and in which phase does it happen?

Answer 10

In the hypothesizing phase

you simply inspect your data and seek for interesting observations/patterns

Question 11

What is confirmation research and in which phase does it happen?

Answer 11

Most common in psychology research.

Requires a hypothesis (h1) to be tested, which is clearly stated before the data is collected.

Question 12

What is HARKing?

Answer 12

• Hypothesizing after results are known.

- Looking for patterns in a database and then formulate a hypothesis afterwards –> order is wrong.

Question 13

How can you avoid HARKing?

Answer 13

Be explicit about your variables

Question 14

How can you be explicit about your independent variables?

Answer 14

• Do not leave out certain conditions, or even exclude participants
• Do not categorize certain variables in a different way
• Do not create new variables based on your data
• Do not measure additional variables that may be analyzed as mediators, moderators, or covariates.

Question 15

How can you be explicit about dependent variables?

Answer 15

• Do not measure one DV in multiple ways

Question 16

Of what should you be aware in question design and in which phase does it happen?

Answer 16

Design phase

Biased questions lead to biased data

Question 17

What are three examples of unexpected biases in question design?

Answer 17

1. Positive vs. negative vs. bipolar questions
2. Response order effects
3. Question order effects

Question 18

What is the conclusion on positive vs. negative vs. bipolar questions?

Answer 18

Positive questions = more negatively
Negative questions = more positively
bipolar = the same as positive

Question 19

What is the conclusion of the classic study by Rugg?

Answer 19

People are more likely to say no to ‘forbid’ questions than to say yes to ‘allow’ questions.

Respondents are more likely to disagree with negative questions than to agree with positive ones

Question 20

What is the conclusion of the response order effects with the study of eyetracking by Galesic et al?

Answer 20

Respondents spends more time looking at the first few options in a list of response options than those at the end of the list.

AND: the longer the respondents looked at the response options, the more likely they were to choose one of them.

Question 21

What can you do best to avoid response order effects?

Answer 21

Best is to present response options in a random order to all participants (programs like qualitrics allow you to do that)

Question 22

What is the questions order effect?

Answer 22

The first question frames the second one, provides the context in which the second question is interpreted.

Question 23

What is the part-whole contrast effect?

Answer 23

When the general and specific items are asked in different orders, the results for the specific items are generally unaffected, whereas those for the general item change significantly

Question 24

What should you avoid in the data collection phase?

Answer 24

• Avoid non-random assignment people to conditions (e.g. females are more sensitive to it then male)
• Avoid incomplete blinding

Question 25

When are participants blinded?

Answer 25

• Participants are blinded if they do not know about the purpose of the study
• Researchers are blinded if they do not know what conditions a participant is in
• Blinding is also important during data coding

Question 26

What is intermediate significance testing?

Answer 26

Collect data –> Analyze data

• Significant effect? Yay, lets publish
• Effect not significant yet? –> Collect data

Question 27

What is type 1 error and in which phase does it happen?

Answer 27

Analysis phase

Claiming there is an effect when in fact there is no effect
- The chance to make this error will be less than 5%

Question 28

What is type 2 error and in which phase does it happen?

Answer 28

Analysis phase

Claiming there is no effect when in fact there is an effect

• According to fisher, the chance to make this error should be around 20%
• Related to the ‘power’ of a test ( 1 - bèta)
• Cannot know actual chance because is based on size of effect and sample size

Question 29

Of what should you pay attention to in the reporting phase?

Answer 29

• Allow others to reproduce your study and analysis
• Allow others to replicate your study
• Do not report ‘HARKed’ findings
• Be honest about everything you report
• Publish both significance and non-significant effects

Just present it the way it is

Question 30

What is a solution for questionable research?

Answer 30

Pre-registration