4 model selection Flashcards
What different kind of criteria does a good model fulfill?
Qualitative
Quantitative
What qualitative criteria does a good model fulfill?
explanatory adequacy (model is plausible & compatible with established knowledge)
interpretability (parameters should be linked to psychological processes/constructs, ideally: parameter estimates show construct validity)
faithfulness (credibility) (ability to describe the data stems from the structural theoretical assumptions in the model)
What quantitative criteria does a good model fulfill?
falsifiability
Goodness-of-fit (distance between observed & predicted data resonably low)
complexity reasonably low (simpler model should be preferred -> less flexible, more precise)
generalizability (fit future data sets sampled from the same population, good model representa systematic trends in data, not noise)
What is overfitting?
Random noise is fitted by a model in addition to systematic variance
What does overfitting mean for the generalizability of the model?
less generalizable as random noise varies across samples of the same population
How does the number of free parameters in a model relate to Overfitting?
more free parameters => overfitting more likely
Which methods try to tackle the problem of overfitting?
Likelihood ratio test (nested models)
AIC & BIC (Akaike/Bayesian Information criterion)
Cross validation
MDL (Minimum Description Length)
Bayesian model selection
What’s the idea behind the likelihood ratio test (nested models) in tackling the problem of overfitting?
create nested model in which some free parameters are fixed
significance test whether model with more free parameters fits better -> if no: use model with less free parameters
What’s the idea behind the AIC & BIC in tackling the problem of overfitting?
correct fit values for number of parameters in the model
more parameters decrease corrected fit
What’s the idea behind Cross-validation in tackling the problem of overfitting?
Data split in half: validation & calibration sample
calibration sample predicts data -> fit in the validation sample -> overfitting if bad
What’s the idea behind MDL in tackling the problem of overfitting?
find the minimal coding length for model + data, taking into account: number of parameters & functional form
What’s the idea behind Bayesian model selection in tackling the problem of overfitting?
rooted in bayesian statistical theory, incorporates flexibility
Describe the idea of construct validation of model parameters.
parameters should represent cognitive Processes
Describe the method of construct validation of model parameters.
experimental manipulations that should only affect specific model parameters
-> convergent & discriminant validity
e.g. manipulate source similarity
Why is construct validation necessary?
interpretability
credibility/plausibility
