L3: Validity Flashcards
Validity - Definition
The degree to which evidence and theory support the interpretation of test scores for proposed uses
What are features of validity? (3)
1) Validity is a property of a psychological test score interpretation
2) A psychological test is valid to a certain degree (it is not binary)
3) Validity is based on both theory and empirical evidence
What are the various types of evidence for validity?
1) Content Validity
2) Response Process Validity
3) Internal Validity (structure)
4) Associative Validity
5) Consequential Validity
Content Validity - Definition
Degree to which the content of a measure truly reflects the full domain of the construct for which it is being used
Who evaluates the content validity of a measure?
Experts
Face Validity - Definition
Degree to which a measure appears to be related to a specific construct in the judgment of non-experts
Who evaluates the face validity of a measure?
Participants / test users
What characteristics of a test reduce content validity?
1) Construct Underrepresentation
2) Construct Irrelevant Content
Response Process Validity - Definition
The match between the psychological processes that respondents actually use when completing a measure & the processes that the researcher intends them to use
Response Process Validity - Evidence
Direct Evidence:
1) Think out loud protocols
2) Interviewing respondents
Indirect Evidence:
1) Process Data (ex: eye tracker)
2) Statistical Analysis (of results)
3) Experimental manipulation of the response process
What is direct evidence for response process validity?
1) Think out loud protocols
2) Interviewing respondents
What is indirect evidence of response process validity?
1) Process Data (ex: eye tracker)
2) Statistical Analysis (of results)
3) Experimental manipulation of the response process
What are general threats to response process validity?
1) Poorly designed items
2) Respondents (guessing, lack of motivation)
Internal Structure Validity - Definition
The match between the actual internal structure of a test & the structure that the test should theoretically possess.
What are features of a test with good structural validity?
1) The nr of factors match the theory
2) Rotated factor loadings display the theoretical structure
3) Factor correlations comply with what is expected based on the theory/literature
What are features of a test with poor structural validity?
1) The nr of factors to not match the theory
2) Rotated factor loadings do not display the theoretical structure
3) Factor correlations do not comply with what is expected based on the theory/literature
Construct Validity - Definition
Degree to which test scores can be accurately interpreted as reflecting a particular construct
Nomological Network - Definition
Network that summarises all theoretical relations between the construct of interest & all other related constructs and variables
A nomological network is used to examine what form of validity?
Associative Validity
Associative Validity - Definition
Match between a measures ACTUAL associations with other measures & the associations that the test SHOULD have with other measures
What are the various parts of a nomological network?
1) Validity coefficients - the individual correlations
2) Convergent Evidence - correlations between constructs that ARE related in theory
3) Discriminant Evidence - lack of correlations between constructs that ARE NOT related in theory
Criterion Validity - Definition
Degree to which test scores are actually related to a particularly important criterion variable that they should be correlated with. Example: GPA & Salary
What are the types of criterion validity?
1) Concurrent Validity
2) Predictive Validity
Concurrent Validity - Definition
Association between the construct & an observed variable AT THE SAME TIME
Predictive Validity - Definition
Association between the construct & an observed variable measured IN THE FUTURE
What is the main challenge when interpreting validity coefficients?
Uncertainty regarding whether the variance in the data is attributable to trait variance or method variance.
Trait Variance - Definition
Shared variance in the test scores due to the same trait/construct
Method Variance - Definition
Shared variance in the data due to the same method
What is the Multitrait-Multimethod Matrices tool used for?
Interpreting validity coefficients
Convergent Validity - Definition
Degree to which test scores are actually correlated with other measures that they SHOULD be correlated with
Discriminant Validity - Definition
Degree to which test scores are actually uncorrelated with other measures that they SHOULD NOT be correlated with
What association is examined under a monotrait-heteromethod correlation?
Same Trait & Different Method
What form of validity is examined under a monotrait-heteromethod correlation?
Convergent Validity
What association is examined under a heterotrait-heteromethod correlation?
Different Trait & Different Method
What form of validity is examined under a heterotrait-heteromethod correlation?
Discriminant Validity
What association is examined under a heterotrait-monomethod correlation?
Different Trait & Same Method
What form of validity is examined under a heterotrait-monomethod correlation?
Discriminant Validity
What correlation is a type of convergent validity?
Monotrait-heteromethod correlation
What correlation is a type of discriminant validity?
1) Heterotrait-monomethod correlation
2) Heterotrait-heteromethod correlation
What is the ideal relation between the correlations in a Multitrait-multimethod matrix?
You want the monotrait-heteromethod correlations to be large, and you want it to be larger than both the heterotrait-heteromethod correlations and the heterotrait-monomethod correlations.
Consequential Validity - Definition
The match between the ACTUAL consequences of using a measure and the consequences that SHOULD be seen
What are the types of evidence of consequential validity?
1) Intended Effects - degree to which the use of the test scores has the intended effects
2) Unintended Differential Impact on Groups - do the test scores end up unintentionally weighted in favour of one group and against another group?
3) Unintended Systemic Effects - does the test have an unintended effect on organisational systems?
What are the main factors that affect validity coefficients?
1) Associations between constructs
2) Random measurement error (& reliability)
3) Restricted range
4) (Differential) skew
5) Method variance (lack of)
6) Prediction based on single vs multiple events
7) Time (between tests)
How do associations between constructs affect validity coefficients?
The greater the association between constructs, the higher the correlation
How does random measurement error affect validity coefficients?
Random measurement error in the data leads to lower reliability & smaller correlations + validity coefficients
How does restricted range affect validity coefficients?
Artificially restricted range will lead to a smaller correlation & validity coefficient
Restricted Range - Definition
When a test does not reflect the true distribution of scores, as it is limited to only a subset of the data. For example, Test A has many people scoring the maximum possible score, while Test B has a normal distribution of test scores.
How does reliability affect validity coefficients?
Lower reliability (due to greater measurement error) will lead to a smaller correlation and validity coefficient.
How does skew affect validity coefficients?
A differential skew will lead to a smaller correlation, and lower validity coefficients.
How does method variance affect validity coefficients?
Method variance does not reduce the predictive validity correlation. Using an identical method will result in a higher correlation, but this will be artificially inflated due to method variance, and therefore not be a reflection of the true correlation. Therefore, the correlations will be more accurate when employing methods that differ from each other.
How does event prediction affect validity coefficients?
- Validity coefficients based on correlations between variables measured at different times (predictive validity correlations) will be smaller.
- Validity coefficients based on correlations between variable measured at a single time point (concurrent validity correlations) will be larger.
What is the potential effect of using predictive validity correlations when estimating validity coefficients?
It might result in an underestimation of validity coefficients
What is the potential effect of using concurrent validity correlations when estimating validity coefficients?
It might result in an overestimation of validity coefficients
How does intervals between testing affect validity coefficients?
Longer time periods between testing will reduce the (predictive) validity correlations
Validity Generalisation - Definition
Process of evaluating a tests validity coefficients across a large set of studies (same constructs, different operationalisations. is the nomological network consistent?)
What is a QVC procedure used for?
It is used to quantify the degree of fit between the predicted pattern of validity correlations & the actual pattern of validity correlations obtained in the data
What are the steps of a QVC procedure?
1) Generate a predicted pattern of validity correlations
2) Gather data & compute actual pattern of validity correlations
3) Examine to what degree the predicted pattern matches the actual pattern
For what kind of validity is a QVC procedure used?
Construct Validity
