Reliability and Validity Flashcards
Types of reliability - reliability across time
· Test-retest reliability
· Involves two administrations of the scale
- Assumes that the construct is stable across time
Types of reliability - internal consistency:
· Split-half reliability
· Cronbach’s alpha
· McDonalds omega hierarchal and omega total
· Involves only one administration of the scale
- How most papers test reliability
- What we will be doing for the lab report
Reliability across time - test-retest:
· Test-retest reliability:
- The consistency of your measurement when it is used under the same conditions with the same participants
- Example procedure: Administer your scale at two separate times for each participant. Compute the correlation between the two scores
· Test-retest reliability doesn’t work if you’re studying a construct that is expected to vary across time points.
· Example: Mood
Internal consistency - split-half:
· Split-half reliability:
- One of the most straightforward ways of testing internal consistency
- You can split a scale into two halves (e.g., a 6-item scale would be split into two sets of 3)
- You calculate a composite score for each half of the scale (e.g., each participant gets a score averaged across the 3 items)
- Calculate the correlation between those two half-scale scores; strong correlation = high split-half reliability
BUT: reliability will depend on exactly how you split the data!
Refining split-half reliability:
· We can refine the split-half method
· We can split the items on the scale every possible way and compute correlations for all splits
· We can obtain an average of all these correlations to give us a sense of the scale’s internal consistency
- This is roughly what Cronbach’s α does!
Interpreting Cronbach’s alpha:
· Interpret as Pearson’s r:
- Varies from 0 (no internal consistency) to 1 (perfect internal consistency)
- No negative values! If you get a negative value, something has gone wrong
· Rule-of-thumb: Acceptable reliability:
· HOWEVER: Depends on the type of construct, progress of the research, etc.
- For cognitive tests, α > 0.8 is appropriate; for other tests α > 0.7 is fine (Kline, 1999)
Other statistics that help interpret Cronbach’s alpha:
· α if item removed
- Calculates α as described but leaves out each item one at a time
- If α improves -> scale is more reliable without it
- Helps identify the worst item (to consider getting rid of it)
· Item-total correlation
- = Correlation between the score on an item and score on the scale as a whole
Cronbach’s alpha - a history of (mis)use:
· Cronbach’s alpha is widely used
- 87% of papers reporting any measure of internal consistency, reported Cronbach’s alpha
- The where Cronbach’s alpha was first introduced (Cronbach, 1951) is one of the most cited English language research articles of any discipline (just under 65000 citations)
- And this is a conservative indicator of its popularity! Many report Cronbach’s alpha without citing the original paper
· Sources of misuse:
- Cronbach’s alpha makes assumptions about the shape of the factor model (i.e., how items relate to factors); if these assumptions are not met, Cronbach’s alpha is misleading
- Using Cronbach’s alpha as evidence of a scale’s dimensionality: a big no-no!
- Cronbach’s alpha is sensitive to the number of items in a scale
The assumptions of alpha - tau-equivalence:
· Alpha assumes tau-equivalence. This corresponds to a factor model with the following features:
- Items have equal loadings
- Items indicate only one factor
- This is unrealistic. Ideally, items have strong primary factor loadings, but they still have loadings on other factors even if they are weak (and even if we ignore them!)
Alpha and unidimensionality:
· Alpha is designed to be computed for unidimensional scales (i.e., scales with only one factor); doesn’t work properly otherwise! But you can’t use it as a measure of unidimensionality
Cronbach’s alpha is sensitive to number of items:
· Having more items in a scale leads to a higher Cronbach‘s alpha, regardless of the actual internal consistency of the scale
· Measures of internal consistency should tell us how strongly items relate to one another: high inter-item correlations indicate high internal consistency. But that‘s not always the case for Cronbach‘s alpha…
· When looking at scales with Cronbach’s alpha of .80, Cortina (1993) found that:
- A scale with 3 items had an average inter-item correlation of .57
- A scale with 10 items had an average inter-item correlation of .28
· In other words: the 10-item scale has worse internal consistency (i.e., low inter-item correlations) than the 3-item scale but Cronbach’s is identical
- You can trick alpha to suggest good internal consistency simply by adding more items to your s
McDonald’s omega:
· Great alternatives to Cronbach’s alpha
· Omega (both total and hierarchical) does NOT assume tau-equivalence or unidimensionality
- Omega uses the factor structure obtained by running a factor analysis (see last week’s lecture!)
· Omega (both total and hierarchical) assume the existence of a general factor
- Usually, not a problem in scale development: we assume that even items belonging to different factors are related! After all, they were all designed to capture the same construct (before we might have learned that our construct is actually a set of related constructs)!
Omega - assumed factor structures:
· Omega hierarchical ωh
- Appropriate for unidimensional scales: items share variance with a general factor
· Omega total ωt
- Appropriate for multidimensional scales: items share variance with both the extracted factors AND the general factor (an overarching, higher order factor
McDonalds omega - testing assumptions:
· Both Omega hierarchical ωh and Omega total ωt assume that there is a general shared factor
· The R output shows you a table of factor loadings where the first column shows each item’s loadings with the general factor
· The second column shows you each item’s loadings for each of the factors extracted in your factor analysis (in this case 4)
· These loadings should be similar to the ones obtained in your factor analysis but they are not identical—unlike your original factor model, this model also includes the general factor
Interpreting omega scores:
· Omega is interpreted in the same way as alpha: 0 (no internal consistency) to 1 (perfect internal consistency)
· Omega hierarchical is smaller than omega total
- Makes sense! We have a multidimensional scale (i.e., more than 1 factor) so the omega hierarchical which tries to fit a single-factor model will
· You should also look at the omega total scores for each of your factors
