Wk 6 - Analytic Strategy Flashcards
What are 3 steps that motivate hypothesis testing?
Desire to experimentally investigate some phenomenon
Measure and/or manipulate a behaviour
Analyse data to determine any effect
What are 3 possible sources of variation in data?
and we can… (x1)
All is systematic
All is random
Both systematic and random factors involved
Dismiss the first in psych…
What is the current standard for evaluating data in psych? (x1)
Which… (x2)
NHST
Distinguishes random chance alone vs chance+systematic effects
Establishes statistical hypothesis of no systematic effects
Give 3 egs of assumptions made by the null hypothesis
But it should be remembered that it… (x1)
No difference in mean scores between groups
No differences in variances across groups
No systematic relationship between variables
Allow for diffs in data - just assumes not meaningful
What does NHST evaluate in practice? (x2)
Probability of observing the data (or more extreme data), if the null hypothesis were true
This probability, p(d|H0), is expressed by p values
If we adopt a conventional p < .05 criterion, significant results imply…(x3)
Ulikely to obtain the data if H0 were true
We reject H0 as viable account - effects not just chance
And endorse account of theoretically interesting/systematic factors
What are the limitations of the inductive leap from data to hypotheses in NHST? (x2)
We evaluate p of obtaining DATA
Then use this to make inferences about HYPOTHESES
What mathematical logic demonstrates the limitations of NHST? (x4)
NHST dosn’t evaluate probability of the null being true
Assumes that it is
The probability of obtaining the data if the null is true,
Is not equal to the probability of obtaining the null given the data
If an AIDS test has a hit rate of 99.99%, and a correct rejection rate of 99.99%,
What is the probability of a positive result if you actually have the disease?
99.9%
If an AIDS test has a hit rate of 99.99%, and a correct rejection rate of 99.99%,
What is the probability of having the disease given a positive test result?
49.98%
What explains the difference between the probability of a positive result if the hypothesis is true,
And the probability of the hypothesis being true given a positive result? (x4)
The overall probability of the hypothesis being true, ie:
1/10 000 men have AIDS
Which gives 1 true, positive test
And 1 false positive (given 9 998 correct rejections
So out of 2 positive results, only 1 actually has AIDS
Why is it critical to remember p(d|H0) ≠ p(H0|d) ? (x3)
p tells us nothing about experimental hypotheses,
Or the probability of replicability
Only about the data we currently have
Why is NHST still relevant, despite legitimate criticisms? (x2, x3)
Not generally just testing for AN effect,
*But contrasting theoretical predictions of where/when they’ll occur
We seldom hypothesise a null
* Predict an event WILL occur, rather than not * Null being true is 1 in a zillion weays of not finding an effect
What are 2 theoretical goals of research?
To contrast different theories of psychological phenomena
Find out what is true about the world and explain it
What are 2 communicative goals of research?
Inform others about how well different theories account for psych phenomena
Convince others about what is true in the world and how it ought to be explained
How do statistics facilitate the theoretical goals of research? (x2)
Quantitatively assess mis/match between theory and data
Allow distinguishing random from systematic variation
How do statistics facilitate the communication goals of research? (x2)
Strengthen evidence supporting theoretical claims
*Objective reference for interpreting implications of data
How do communication goals influence research design? (x2)
Readers/publishers have expectations re design/analysis
Need to think on how research question influences study design and how variables measured, analysed, interpreted
When should you be deciding on data analysis plan? (x3)
Well before collecting data:
* Can constrain method, and therefore viability question * Protect against collecting unanalysable data
What are the practical benefits of planning analysis before collecting data? (x2)
Organise thinking about your data set a priori
And about project/how to answer research questions
What are the 4 steps in developing data analysis plan/deciding on key statistical tests to address research questions?
Review:
*Research Questions
*Study Design and Measurement
*Hypotheses
Choose appropriate statistical testsWhat are the considerations when reviewing research questions for data analysis planning? (x1, x2)
What constructs are you interested in and how are they related?
If there are multiple IVs, are some more important than others?
What is your contribution to existing research?
What are the considerations when reviewing study design and measurement for data analysis planning? (x3, x3, x3)
Were your IVs just measured:
* Single/multiple time points? * Continuous/categorical scales?
Or manipulated?
* Number of levels of IV? * Between/within Ps?
Relevant DVs:
* 1 or more time points? * Continuous/categorical?
What are the considerations when reviewing hypotheses for data analysis planning? (x1, x1, x2, x2)
What are operationalised IV and DVs?
Theoretically driven main effects of IVs on 1+ DVs?
Expected diffs across conditions?
*Interactions?
Follow-up comparisons req'd?
*Planned vs post hocWhat are the considerations whenchoosing appropriate statistical tests during data analysis planning? (x2, x1, x1, x1)
Diff methods apply to diff designs
*SPSS will allow anything!
Between-/within-Ps, cross-sectional or longitudinal?
How many IVs/DVs? Continuous/categorical? Covariates?
Mediation/moderation hypotheses?
What are the diffs between practical and theoretical implications of experimental findings? (x2)
So it’s good to… (x1)
Reporting p values is expected,
But can be poor indicator of importance
Report effect size too (eg r-squared)
Why does effect size matter?
Any effect is significant with big enough sample
Need sufficient power to find effects of expected size
What 3 factors influence statistical power?
Significance level (alpha) Sample size Error variance
What can we do to ensure adequate sample size/power? (x3)
Bigger enables detection of smaller effects
Can use Power Analysis to calculate
Decide BEFORE commencing study
What design/analytic techniques can be used to maximise power? (x2)
Use blocking designs
Include control variables (eg ANCOVA)
Simpler is better when communicating research results, so you should… (x3, x3)
Limit number of IVs/DVs
- Easier for you/others to make sense of data
* And to write compelling narrative
Use simplest statistical analyses you can
* Don't over-complicate results * Doing so creates suspicion
Wneh seeking simplicity in communicating results, its important to remember that…(x4)
You can’t conceal inconvenient info!
* Note any missing data issues * Report any preliminary analyses, eg EFA * Be clear on details of analyses
What is HARKing? (x1)
Which entails? (x2)
Hypothesising after results are known
Revising hypotheses post-analysis
Presenting post-hoc hyps as if a priori
What is the general hypothetico-deductive scientific method? (x4)
Develop theory-driven, testable hypotheses
Design experiment
Report results
Draw conclusions based on evidence
Why is post hoc HARKing not science? (x2)
By definition, unfalsifiable
*Hypotheses tailored to fit data
Explain why are HARKed hypotheses bad? (x3)
Often begins with no EXPECTED effects in data, but something UNexpected observed
No theoretical motivation to explain this - likely a false positive
Buckets of replication failures needed to discredit initial ‘finding’
If HARKing is so bad, why does it happen? (x2, x3)
Researchers want to tell compelling story
*Often post-hoc theorising plays out subconsciously
Meta-scientific reasons
* Publish or perish * Journals don't publish null findings
How to recognise HARKING: It probably is if… (x2)
You acknowledge lack of a priori hypotheses, but draw post-hoc conclusions from data
Fail to report unsupported a priori hypotheses
How to recognise HARKing: It’s not if… (x1)
Competing a priori hypotheses presented, then conclusions drawn from data
What are 3 ways to spot harking? (x2, x2, x1)
Mismatched prior theory and a priori hypotheses
*Incoherent, overly complex, implausible
Mismatched research questions and methodology
*Odd study setup
Unmotivated/expected testing new condition/qualifier that conveniently fits the data
What are 4 ways to stop yourself from HARKing?
Pre-register studies
Report what you do
Be honest about hypotheses
Use multiple studies to examine unexpected results
