Midterm 1 - Validity (Ch. 5 and Miller et al.) Flashcards
MTMM matrices are particularly useful bc they provide us with ______ coefficients, ____ coefficients, and _____ coefficients
convergent validity, discriminant validity, reliability
what is the difference btw convergent and discriminant validity evidence?
convergent: degree to which data from two measures targeting the same construct agree (or correlate)
discriminant: degree to which data from two measures targeting distinct constructs disagree (or do not correlate)
In the MTMM, validity diagonals consist of validity coefficients depicting the correlations between _____ and _____
the same trait measured using distinct methods (monotrait-heteromethod values)
In the MTMM, discriminant validity coefficients are organized into (hetero/mono)trait-(hetero/mono)method triangles which depict correlations among ________
heterotrait-monomethod and heterotrait-heteromethod triangles
HTMM: corr. btw measures of diff constructs using same method
HTHM: corr. btw measures of diff constructs using diff methods
In the MTMM, reliability coefficients are computed by ____
not calculated in the matrix, just placed where monotrait-monomethod coefficients would go to depict relative reliability of each measure
Validity assesses the extent to which a test _____
measures the attribute/construct it is designed to measure
Validity is not a yes/no decision. It comes in ___
degrees (no specific range of coefficients)
generating validity evidence is ____
an ongoing process
What are the 3 types of validity?
- content validity
- criterion validity
- construct validity
What is face validity?
- 4th type of validity but not considered actual/sufficient form of validity
- whether test appears to measure what it is supposed to measure
- can help ppl trust the test and take it seriously BUT can also run into issues w responses based on social desirability
Content validity evaluates adequacy of ____
domain representation
What are the 3 steps of establishing content validity?
- describe content domain (identify boundaries, determine structure-eg proportion of test on a given topic)
- inspect test
- form judgement that test measures what it is supposed to measure
Content validity is high when the test has good _____ and when _____
good content coverage (representative of entire domain) AND relevant items (inside content domain)
Why is content validity not sufficient to decide a test is valid?
no info ab relation of test to external constructs or external variables!
Criterion validity evaluates effectiveness of the test in _____
predicting narrowly and specifically identified outcomes/criteria
What is a criterion? Is it objective or subjective?
- standard researchers use to measure outcomes such as performance or attitude
- can be objective (observable and measurable) or subjective (based on a person’s judgement)
What are the 2 subtypes of criterion validity?
- concurrent validity: criterion available at same time as test
- predictive validity: criterion measure available in the future
What are the differences between concurrent and predictive validity?
- concurrent: large group takes test and another measure (criterion) AT SAME POINT IN TIME
- predictive: large group takes test and scores are held, criterion measure is administered LATER
What is the impact on validity if…
…a criterion measures fewer dimensions than those measured by the test?
…a criterion measures more dimensions than those measured by the test?
- decreased evidence of validity
- criterion contamination (also decreased validity)
What is a validity coefficient?
correlation btw test and criterion
validity coefficients are rarely greater than ____
.60
(T/F) correlation is a measure of effect size
TRUE
What are 3 factors limiting validity coefficients?
- restricted range of scores decreases validity coefficients
- low reliability of test scores decreases validity coefficients
- low reliability of criterion decreases validity coefficients
What is the formula to correct for unreliability in test?
rxycorr = rxy / √rxx’
rxycorr = estimated validity coefficient
rxy = observed validity coefficient (of unreliable measure)
rxx’ = reliability coefficient of predictor test
What is the formula to correct for unreliability in criterion?
rycorrxcorr = ryx / √(rxx’*ryy’)
rycorrxcorr = estimated validity coefficient
ryx = observed validity coefficient (of criterion)
rxx’ = reliability coefficient of predictor test
ryy’ = reliability coefficient of criterion
What is an issue with the formula to correct for unreliability in criterion?
can sometimes generate unrealistic numbers (eg above 1)
Why can small validity coefficients still have practical utility?
- can use to compare validity between different types of tests
- ex study that compared validity of psych vs medical tests
The process of construct validation involves accumulating information indicating that _____
the test acts as it is supposed to act given the construct definition
What are the 3 steps of construct explication? Together, these steps help to establish a _______
- identify behaviours that relate to construct
- identify other related constructs
- identify behaviours related to other constructs
nomological network!
What are the 6 steps to gather psychometric evidence of construct validity?
- evidence of validity based on content
- evidence of validity based on relations w criteria
- reliability
- experimental interventions
- convergent evidence of validity
- discriminant evidence of validity
The nomological network consists of ____, ____, and ____
constructs, their observable manifestations, and the relations within and between constructs and their observable manifestations
In examining evidence of validity based on content, we want to make sure there is no _____ and no ____
no construct underrepresentation
no irrelevant construct representation
In examining evidence of validity based on relations w criteria, we are looking to see if _____
the relations of the test w external criteria are as would be expected based on theory
In examining evidence of validity based on reliability of the test, we need to make sure that ______ is not too (high/low)
test-retest or internal consistency is not too high or low given the construct
In examining evidence of validity based on experimental interventions, we are checking to see that _____
situational changes that should influence the test scores DO influence test scores (eg starting anxiety meds should reduce score on anxiety measure)
What is the difference between convergent and discriminant validity?
- convergent: test scores correlate w other measures of same construct OR measures of constructs to which test should be related (think nomological net)
- divergent: test scores UNcorrelated w measures of constructs to which the construct should not be related (think nomological net)
method variance is a form of ___ error
systematic
MTMM allows us to examine ___, ___, and ___
divergent validity, convergent validity, and method variance
In using the MTMM, we are looking for:
- (high/low) trait variance (variance due to construct)
- (high/low) method variance
- (high/low) irrelevant variance (variance shared w unrelated measures)
- HIGH trait variance
- LOW method variance
- LOW irrelevant variance
What are the minimum requirements to run a MTMM?
at least 2 measures representing 2 diff methods for each of 2 constructs
In the MTMM, monomethod blocks are located _____. They contain ______ and _____.
along the main diagonal
contain reliability diagonal (middle of table) and heterotrait-monomethod triangles
What does the reliability diagonal tell us?
- general reliability of test (eg test-retest,, internal)
- can use these to compare relative reliability of each
What do heterotrait-monomethod triangles tell us? Do we want these numbers to be high or low?
- tell us if there is method variance
- should be low (if high, means there is method variance)
- generally should be lower than reliability diagonal
In the MTMM, heteromethod blocks are located ____. They contain ____ and ____.
at bottom left of table (in an L)
contain validity diagonals and heterotrait-heteromethod triangles (lying on each side of validity diagonals)
What do the validity diagonals tell us? Do we want these numbers to be high or low?
- they are monotrait-heteromethod
- tells us correlation of same trait measured w diff methods
- we want this to be as high as possible (convergent validity)
What do the heterotrait-heteromethod triangles tell us? DO we want these numbers to be high or low?
- can use as evidence of divergent validity
- want to be as low (close to 0) as possible!
How can you assess which method has less method variance using MTMM?
- look at heterotrait-monomethod triangles
- avg each triangle (method)
- lowest avg has least method variance
Correlations btw diff constructs using same method should be (higher/lower) than convergent validity coefficients (validity diagonals)
LOWER
How can you assess convergent validity of constructs using MTMM?
- look @ validity diagonals
- avg of 3 correlations for each construct
- highest avg has best convergent validity
What are 3 advantages of the MTMM approach?
- allows examining convergent and discriminant validity simultaneously
- stresses the characteristics that a good test should have
- reflects nature of construct validation (can’t just use a single coefficient!)
What is a disadvantage of the MTMM approach?
- sometimes not feasible (high demands)
(T/F) a test can be reliable without being valid
TRUE
