L4. reliability- empirical estimates and importance

Question 1

Cronbach’s alpha assumptions
1. essential tau equivalence
2. error terms
3. error scores
4. unidimensionality

Answer 1

essential Tau-equivalence
- This implies that each item is an equally strong indicator of the true score scores, but they may differ by a constant.
- the items can have different means.

error terms
- That each item’s error term is uncorrelated with every other item’s error term.
- Sometimes you will find two items within a test that are more similar to each other than the other items in the test.
- This often results in a positive correlation between the “error” terms associated with these two items.
- Cronbach’s alpha can not take these positively correlated error terms into account, so it is assumed that they do not exist.

error scores
- That the error scores are uncorrelated with the true scores.

unidimensionality
-That the items used to generate a composite score measure only one attribute or construct.

Question 2

standardised coefficient alpha, when should we use it?

Answer 2

• Standardized coefficient alpha should be applied to scores that have been converted from raw scores to standardized scores.
• For example, if you had z-scores and you wanted to calculate the level of internal consistency associated with a composite which consisted of a sum of two or more z-scores, you would use the standardized version of coefficient alpha.
• In practice, we don’t usually analyse standardized scores, but it does happen from time to time.

Question 3

kudar richardson

Answer 3

• formula was introduced to estimate the internal consistency reliability associated with composite scores based on dichotomously scored items.
• cronbachs alpha is extension of kudar

Question 4

Types of reliability assumption models
parallel
tau
essentially tau
congeneric

Answer 4

Parallel:
- Equal true score variance, equal means, equal error variance
- very strict only exists in theory

Tau-Equivalent:
- Equal true score variance, equal means, unequal error variances
- don’t need to be psychometrically equivalent, different error contributes tot hem

Essentially Tau-Equivalent:
- Equal true score variance, unequal means, unequal error variances
- assumptions for Cronbachs
- real-world data

Congeneric:
- Unequal true score variance (but all greater than zero), unequal means, unequal error variances
- real world data

Question 5

Cronbach’s alpha a lower-bound estimate
cronbachs overestimations

Answer 5

• the assumption of essential tau-equivalence is rarely satisfied,
• therefore Cronbach’s alpha will tend to underestimate reliability.
• 10 or more items the underestimate is in the maximum order of about .02.
• violating the assumption of non-correlated error terms can result in Cronbach’s alpha overestimates in the order of .05 to .06.

Question 6

Factors affecting Cronbach’s reliability
number of items
sample homogeneity
sample generalisation

Answer 6

Number of items
- from 2-8 items alpha rises rapidly, after 15 it increases gradually
- this effect is greater when inter-item corr is greater

Homogeneity
- more homogenous sample will yield lower reliability estimates than a heterogeneous sample.
- greater homogeneity implies less variance.
- Less variance implies smaller inter-item correlations, all other things equal.

sample generalisation
- reliability is measure across samples and see how much they stay the same, they will if scores are reliable eg no meaningful difference in means
- For reliability of .70, a sample size of 400 is required for respectable confidence.
- For reliability of .90, a sample size of 100 is required for respectable confidence.

Question 7

coefficient omega

Answer 7

Coefficient omega (ω) is a more modern approach to estimating internal consistency reliability.
does not assume tau equivalence
Coefficient omega is a better estimate of internal consistency reliability
Coefficient omega is based on factor analysis.
Coefficient omega is standardized, therefore, it does not assume that each variable is measured with the same variance.

Question 8

point estimate
standard error
confidence interval

Answer 8

• point estimate is the score a person receives on the test
• if it’s used to determine something in practice, the confidence of this point estimate must be 95% +
• the standard error represents the amount of error around a point estimate in SD form
• the confidence interval reflects a range of values that is likely for the true score to fall in
• can be an estimate for coeff alpha
• OVERLAP = tells both SAMPLES likely come from the SAME POPULATION (but if there is a gap between the 95% CI likely to be diff populations)

Question 9

two types of correlations:
observed score Corr
True score Corr
-> r max
-> Correction for disattenuation

Answer 9

Observed score correlation
- The correlation you get based on the data you have.
- compromised (or attenuated) to the degree of measurement error in your data.
- max corr is less than 1.0

True score correlation
- A hypothetical correlation you can estimate if you know the reliabilities associated with the scores.
- Not compromised by measurement error.

R-max
- max correlation possible given measurement error and reliability
- square root of the product of two tests reliability scores
- Used for finding the correlation if measures had perfectly reliable scores
- VALIDITY COEFFICIENT CANNOT EXCEED the RELIABILITY INDEX

correction for attenuation (disattenuated r)
- max corr when corrected from measurement error
- hypothetical
- the ratio of observed score correlation to R-max
- can go above 1.0
- only perform if observed score corr is significant