Misleading Data Flashcards
(30 cards)
What is publication bias?
The fact that in many fields only papers with significant results are published, and papers without will not be published.
Bias towards significant effects
How can you also call publication bias?
Filedrawer problem
What are the 7 deadly sins of psychology?
- Confirmation bias
- Hidden flexibility | data tuning
- Unreliability | replication crisis
- Data hoarding | Failure to share data
- Corruptibility
- Internment
- Bean counting | publication counting
What is definition of confirmation bias?
The tendency that people have to embrace information that supports their beliefs and reject information that contradicts their believes
In Peter Wason’s Selection Task which cards does the people turn over to test the truth of the rule?
the rule was: if a card has a vowel on one side, then it has an even number on the other side.
E K 2 3
Need to turn over E and 3.
Which card in Peter Wanson’s selection task did we turn over often?
E K 2 3
we turned over 2 often
What could be confirmed with that we turn over 2 often?
We tend to look for confirmation (turning the 2) instead of falsification (turning the 3).
With 3 we disconfirm the rule but that is something we rather not do.
What is null hypothesis significance testing?
- H0 and H1
- P = the probability of an effect if H0 were true but NOT the probability of H- itself being true
if P > 0.05 –> H1 is rejected BUT H0 cannot be accepted.
- inherent bias towards confirming an effect. You cannot confirm that there is no effect.
What are the degrees of freedom in psychological research according to Jelte Wichers?
- Observe (literature)
- Formulate hypothesis
- Prediction (design)
- Test (Data collection & analysis)
- evaluate (present)
What is exploratory research and in which phase does it happen?
In the hypothesizing phase
you simply inspect your data and seek for interesting observations/patterns
What is confirmation research and in which phase does it happen?
Most common in psychology research.
Requires a hypothesis (h1) to be tested, which is clearly stated before the data is collected.
What is HARKing?
- Hypothesizing after results are known.
- Looking for patterns in a database and then formulate a hypothesis afterwards –> order is wrong.
How can you avoid HARKing?
Be explicit about your variables
How can you be explicit about your independent variables?
- Do not leave out certain conditions, or even exclude participants
- Do not categorize certain variables in a different way
- Do not create new variables based on your data
- Do not measure additional variables that may be analyzed as mediators, moderators, or covariates.
How can you be explicit about dependent variables?
- Do not measure one DV in multiple ways
Of what should you be aware in question design and in which phase does it happen?
Design phase
Biased questions lead to biased data
What are three examples of unexpected biases in question design?
- Positive vs. negative vs. bipolar questions
- Response order effects
- Question order effects
What is the conclusion on positive vs. negative vs. bipolar questions?
Positive questions = more negatively
Negative questions = more positively
bipolar = the same as positive
What is the conclusion of the classic study by Rugg?
People are more likely to say no to ‘forbid’ questions than to say yes to ‘allow’ questions.
Respondents are more likely to disagree with negative questions than to agree with positive ones
What is the conclusion of the response order effects with the study of eyetracking by Galesic et al?
Respondents spends more time looking at the first few options in a list of response options than those at the end of the list.
AND: the longer the respondents looked at the response options, the more likely they were to choose one of them.
What can you do best to avoid response order effects?
Best is to present response options in a random order to all participants (programs like qualitrics allow you to do that)
What is the questions order effect?
The first question frames the second one, provides the context in which the second question is interpreted.
What is the part-whole contrast effect?
When the general and specific items are asked in different orders, the results for the specific items are generally unaffected, whereas those for the general item change significantly
What should you avoid in the data collection phase?
- Avoid non-random assignment people to conditions (e.g. females are more sensitive to it then male)
- Avoid incomplete blinding
