Model evaluation and violations

Question 1

Model evaluation:

Answer 1

“Models are devices that connect theories to data. A model is an instantiation of a theory […]” (Rouder et al., 2016, p. 2)
– Violations of model assumptions reduce reliability.
– Possibility leading to incorrect conclusions about data.
– Violations don’t render model results wrong; neither are models without violations necessarily correct.
– Generally we need to be cautious with and conscious of violations when interpreting results.

Question 2

Assumptions of parametric models

Answer 2

–	e.g. t-test, ANOVA, linear regression
–	Remember LINE
–	Linearity (for continuous predictor variables)
–	Independence
–	Normality
–	Equal of variance (also homogeneity)

Question 3

Using residuals to evaluate models:

Answer 3

– Linearity: linear relationship between outcome and predictor(s)
– Independence of residuals (remember iid)
– Normality of residuals
– Equality of variance of residuals

Question 4

What are residuals?

Answer 4

Residuals are the unexplained (residual) variance: error in the modelling results.
Distance between observed (y) and predicted rt (y ̂): ϵ=y-y ̂
The closer the residuals are to 0, the lower the prediction error.

– Distributed around 0
– Right / positive skew shows some violation of normality assumption
– rts are 0 bound and known to have a heavy right tail (see e.g. Baayen, 2008)

Question 5

What can we do about positive skew:

Answer 5

– Logarithmic transformation is routinely used in the literature, especially for rts, to correct positive skew (see e.g. Baayen, 2008).
Can also be used for non-linear relationship.

Question 6

Linear / arithmetic scale

Answer 6

Distances between adjacent values must be the same, i.e. ±1 on a linear scale.
Distances between 1 inch and 2 inch is 1 etc.

Question 7

Logarithmic scale

Answer 7

Distance between units is going down as values go up.
	Granularity for small while also including large numbers.
	Basis 10 (⋅ or ÷ by 10): 0.1, 1, 10, 100

Question 8

skewness vs kurtosis

Answer 8

– Skewness is about horizontal shape of the distribution
– Kurtosis is about vertical shape of the distribution

Normality: Kurtosis= the sharpness of the peak of a frequency-distribution curve.

Question 9

Independence of residuals

Answer 9

– Plot residuals across ppt id, predictor, predicted data
– What are we looking for?
– Dependencies:
• linearity, independence violations
• For any value on the x-axis, mean of residuals should be roughly 0.
– Equality of variance:
• For any value on the x-axis, the spread of the residuals should be about the same (constant variance).

Question 10

Equality of variance

Answer 10

Homogeneity (same) vs heterogeneity (different)
Variance = SD^2: deviations between observations and their mean
More diverse groups have a larger variance

Question 11

Linearity

Answer 11

– Linear relationship between outcome and predictor variable
– Relationship can be described with a straight line
– Increase is additive.

Question 12

Linearity: continuous predictor

Answer 12

– Linearity: mean of residuals should be 0 at any age.
– Divided age into 20 equal sized bins.
– Calculated the mean of the residuals for each bin.