6. Reliability and external validity (in scale development) Flashcards
Factor analysis allows us to test whether our s____ captures one c____ (factor) or s___ r____ c____ (factors)
scale, construct, several related constructs
Reliability across time is measured using t____-r____ reliability. This involves t____ administrations of the scale and assumes that the c____ is s____ across time
test-retest
two, construct, stable
Internal consistency is measured using:
1. S____-h____ reliability
2. C____ a____
3. M____ o____ h____ and o____ total
Involves only o____ administration of the scale is about item h____. This is how most paper test reliability.
- Split-half
- Cronbach’s alpha
- McDonald’s omega hierarchical and omega total
one, homogeneity
Test-retest reliability is the c____ of a measure when used under the s____ c____ with the s____ p____
consistency, same conditions, same participants
Split-half reliability is one of the most straightforward ways of testing i____ c____ (IC). IC is about item h____ (different subsets of items capture the same thing). You can split a scale into two halves, calculate an a____ score for each half of the scale and calculate the c____ between those two half-scale scores.
Internal consistency
homogeneity
average, correlation
strong correlation = high split-half reliability
BUT reliability will depend on exactly how you split the data!
You can refine the split-half method by splitting the items on the scale e____ p____ w____ and compute correlations for all splits. You can obtain an average of all these correlations to give a sense of the scale’s i____ c____. This is roughly what C____’s a does.
every possible way
internal consistency
Cronbach’s a
You interpret Cronbach’s alpha as P____’s r. Varies from __ to __ with no n____ values.
Pearson’s r
0 to 1 negative
Cronbach’s alpha is sensitive to the n____ of i____ in a s____
number of items in a scale
Alpha assumes t____-e____. This corresponds to a factor model with the following features:
1. Items have e____ l____
2. Items indicate o____ o____ f____
tau-equivalence
1. equal loadings
2. only one factor
This is unrealistic. Ideally, items have strong primary factor loadings, but they still have loadings on other factors even if they are weak (and even if we ignore them!)
Alpha is designed to be computed for u____ scales - it doesn’t work properly otherwise!
In a scale with one factor, we can compute a t____ alpha score. In a scale with two factors, we cannot compute a t____ alpha score, but we can compute s____ alpha scores for each factor.
unidimensional
total
total, separate
Having more items in a scale leads to a higher Cronbach’s alpha, regardless of the a____ i____ c____ of the scale. Measures of internal consistency should tell us how s____ items r____ to one another; high i____-i____ correlations indicate higher internal consistency, but that’s not always the case for Cronbach‘s alpha…
You can trick alpha to suggest good internal consistency simply by a____ m____ i____ to your scale
actual internal consistency
strongly, relate, inter-item
adding more items
McDonald’s (1978, 1999) Omega hierarchical ωh and Omega total ωt are great alternatives to Cronbach’s alpha because Omega (both t____ and h____) does not assume t____-e____ or u____. Omega uses the f____ s____ obtained by running a f____ a____
total, hierarchical
tau-equivalence, unidimensionality
factor structure, factor analysis
Omega (both t____ and h____) assume the existence of a g____ f____ (an o____, h____ o____ factor)
total, hierarchical
general factor
overarching, higher order
Usually, not a problem in scale development: we assume that even items belonging to different factors are related! After all, they were all designed to capture the same construct (before we might have learned that our construct is actually a set of related constructs)
Omega hierarchical ωh is appropriate for u___ scales: items share v____ with a g____ factor
unidimensional
variance, general
Omega total ωt is appropriate for m____ scales: items share v____ with both the e____ factors and the g____ factor
multidimensional, variance, extracted, general
Omega is interpreted in the same way as alpha: __ (n____ internal consistency) to __ (p____ internal consistency)
0, no
1, perfect
Both Cronbach’s and McDonald’s omega (both h____ and t____) assume that your items are all coded in the s____ d____. In other words, having a high score on your sale items means the s____ t____ (e.g. having high levels of the target construct).
R____-p____ items measure the same idea but in o____ d____
hierarchical, total
same direction
same thing
Reverse-phrased, opposite directions
Composite score = a single score obtained by a____ (e.g. summing, averaging) the items from a scale. This is the s____ score, intended to represent a pts level of the t____ c____. This is what we use when testing relationships between our m____ with other m____ and, based in this, we infer relationships between our t____ c____
aggregating
scale, target construct
measure, target constructs
A scale may capture o____ construct or s____ r____ c____ (factors). We need to separate c____ scores for each factor. Each factors c____ score will be obtained by a____ the items belonging to each factor
one, several related constructs
composite
composite, aggregating (e.g. summing, averaging)
External validity means how does our scale compare to other e____ scales and is sometimes called c____ validity. There are two subtypes:
1. C____ validity - tests whether constructs that should be rated in t____ are related in r____
2. D____ validity - tests whether constructs hat should not be related in t____ are not related in r____
existing, construct
1. convergent, theory, reality (i.e. their measures correlate)
2. Discriminant, theory, reality (i.e. their measures don’t correlate
Two steps to convergent and discriminant validity:
1. T____ considerations
2. Testing r____
–> Do measures of r____ c____ correlate with your new scale?
–>Do measures of u____ c____ correlate (positively or negatively) with your new scale?
- Theoretical
- relationships
–> related constructs
–> unrelated constructs
