Common Problems in Research Flashcards
What are the common problems in research?
Insufficient power
Non-significant results
Intervention effects
Restricted ranges
Violating the independence of observations
Mistaking correlation for causation
Unequal groups
Intra-group dependency
External validity issues
What is power in research? What is the desired level of power in research?
The probability of capturing an effect. Power ranges from 0 to 1 and researchers aim for a power of 0.8
How is power determined in a study?
Effect size (Size of the effect in the population)
Alpha level (probability of a type 1 error)
Sample size (larger sample size will increase power)
How can power be increased?
Increase alpha level (not recommended)
Minimize number of dependent variables
Reduce error
Increasing size of treatment effect
Recruit correct number of participants for statistical test
How can error be reduced?
A major source of error derives from unreliable measurements. If possible use only measurement instruments with internal reliability of 0.8+
Reducing error increases effect size
How can size of treatment effect be increased?
Increase size of treatment effect by increasing impact of treatment.
Review relevant literature to determine ways of optimizing impact of intervention.
What is the danger with too large a sample size?
Study will be overpowered and risk of capturing trivial population effects of no practical or clinical importance.
Population needs to be enough for 80% chance of detecting real population effects
What is the mistake commonly made with interpreting insignificant results?
Confusing lack of evidence with evidence of no effect.
Failure to find statstically significant effect means:
Effect does not exist and sample data confirms this
Effect does exist but study has insufficient power to detect this
What is the effect of intervention?
Failing to discriminate between intervention effects at the group level and at single-subject level.
A statistically significant effect to the group does not mean every person benefited.
What problem arises from restricted ranges?
If you restrict the range the sample correlation becomes attenuated such that it underestimates the true population value. (eg anxiety vs performance) Imagine a
What are the 3 variables that make one variable cause another?
Covariation - 2 variables must correlate
Directionality - the presumed cause must precede the presumed effect
Causal closure - All other variables that might influence the relationship between the 2 variables must be eliminated or controlled
What is the independence of observations?
States that each score on the dependent variable must be independent of another score. (i.e 2 people completing a questionnaire should do so without interacting with each other)
What happens if the independence of data is violated?
Creates inflated type 1 error rate.
What causes unequal groups?
2 groups are equivalent if they are matched on all extraneous variables that might otherwise produce between-group differences on the dependent variable.
If 2 groups are not equivalent, this is because between-group differences in the dependent variable cannot be attributed solely to the independent variable.
How can group equivalence be achieved?
Randomizing the allocation to groups
