Lecture 10: More on Validity

Question 1

what is the most important evidence of validity?

Answer 1

associations with other variables

Question 2

validity

Answer 2

the degree to which evidence and theory support the interpretations of test scores for proposed uses of a test; , all the content and all of the theory we think we’re measuring

Question 3

What is associations with other variables? what is evidence of this?

Answer 3

does the test’s actual associations with other measures match the associations that it should have with those measures? We do this by assessing: convergence evidence (concurrent/predictive) & discriminant evidence

Question 4

What are key issues in examining and interpreting convergent and discriminant evidence?

Answer 4

1. Methods for evaluating “associations with other variables” 2. Factors affecting observed associations 3. Interpreting associations)

Question 5

Methods for evaluating “associations with other variables”

Answer 5

1. Focused examinations (few criterion variables) 2. Unsystematic examinations of sets of correlations 3. Multi-trait multi-method Matrix (MTMM) 4. Systematic examination of sets of correlations—“Quantifying Construct Validity”

Question 6

Focused Examination

Answer 6

one (or very few) criterion has strong relevance for the implications/meaning of test scores (one or view criterion variables that have an association with the construct; SAT scores predict GPA); validity generalization: process of evaluating a test’s validity over a large set of studies (meta analysis idea)

Question 7

validity generalization

Answer 7

process of evaluating a test’s validity over a large set of studies (meta analysis idea)

Question 8

Unsystematic Examination of Sets of Correlations

Answer 8

several criterion variables (other measures) are examined; probably the most common; “eyeball” the pattern of correlations and draw conclusions regarding convergent and discriminant validity

Question 9

Multitrait-Multimethod Matrix

Answer 9

several criterion variables (other measures) are examined; more systematically evaluated the pattern of correlations and draw conclusions regarding convergent and discriminant validity; not as common, but golden child of evidence

Question 10

Multitrait-Multimethod Matrix steps

Answer 10

1. Measure multiple traits/constructs 2. Use multiple methods of measurement 3. Use each method to measure each construct 4. Compute all correlations and evaluate their pattern); example: validity of measure adult playfulness: constructs 1. Adult playfulness 2. Spontaneity 3. Boredom multiple methods for assessing each construct 1. Self-report 2. Spouse report 3. Observation

Question 11

What are the strongest MTMMM correlation? (ideally)

Answer 11

ideally mono-trait hetero-method correlations are stronger than hetero-trait correlations

Question 12

Quantifying Construct Validity

Answer 12

: several criterion variables are examined; predict correlations to other construct ahead of time and compare to predictions (pretty new, Dr. Davis doesn’t even mention it); systematically evaluate the pattern of correlations and draw conclusions regarding convergent and discriminant validity

Question 13

How to do Quantifying Construct Validity

Answer 13

1. Make concrete expectations for pattern of correlations 2. Quantify match between expected pattern and actual pattern of correlations (How close is the match? Is there a significant degree of match?) **don’t need to know how to do it

Question 14

key issues in examining and interpreting convergent and discriminant evidence

Answer 14

methods for evaluating “associations with other variables”; factors affecting observed associations (why would a validity correlation be high or low); interpreting associations: gauging the size/ meaningfulness of a validity coefficient

Question 15

Factors Affecting Observed Associations

Answer 15

1. Associations between constructs: for example, true score correlation from CTT 2. (Im)precision of measurement (essentially the reliably could impact validity measurement): for example, measurement error / reliability from CTT 3. Restricted Range: significant impacts the correlation of the validity coefficient 4. Relative proportions of differential skew (skewed distributions impact correlations, especially if the skews of two variables are different it results in a lower correlation); if your samples of your measured construct are not normally distributed, it will impact the correlation 5. Method variance: generally, variables measured by different methods are less strongly correlated than other variables measured by the same method 6. Time: generally, variables measured at different times are less strongly correlated than variables measured at the same time 7. Predictions of single event: generally, behavior/outcomes assessed at a single occasion are less predictable than behaviors/outcomes that are aggregated across occasions

Question 16

Interpreting Validity Coefficients

Answer 16

correlation coefficient ranging from 0-1; don’t expect huge validity coefficients; squaring the validity coefficient tells the amount of variance in the criterion that can be explained by the predictor (ex. Using a test of cognitive ability to predict results in a validity coefficient of 0.5. We can say 25% of the variance of the job performance is accounted for by the cognitive ability.); there are a lot of other factors that go into one single measure, we can help by discussing the variance explained

Question 17

Interpreting Associations

Answer 17

1. Squared correlations and “variance explained” 2. Estimating practical effects: binomial effect size display, Taylor-Russel tables, utility analysis, and sensitivity/specificity 3. Guidelines or norms for a field 4. Statistical significance
   * only really need to know about Taylor-Russel tables because they are associated with the base rate; tells us how well our tests can predict future outcomes

Question 18

Methods Evaluating Convergent and Discriminant Validity (Construct Validity)–Book

Answer 18

1. Focused Associations—focus on a few highly relevant criterion variables (Validity generalization—validity coefficients across a large set of studies)
   a. Good for GPA and SAT
2. Set of correlations (correlation between the test and measures of many criterion variables—eyeball the correlations to make judgement on the basis of nomological network)
3. Multitrait-Multimethod Matrices—set clear guidelines for evaluating convergent and discriminant validity based on trait and method variance (MTMMM; monotrait-heteromethod shows convergent)
4. Quantifying construct validity (QCV)
   a. Clear prediction
   b. Collect data and compute actual convergent and discriminant validity correlations
   c. Quantify the pattern of convergent and discriminant validity
   i. Effect size
   ii. Significance test

Question 19

Factors Affecting Validity Coefficient (Book)

Answer 19

1. Associations between constructs (true association between constructs)
2. Random measurement error and reliability; (Im)precise measurement (random error attenuates correlations between tests)
3. Restricted range (correlation can be reduced if the variability in one or both distributions is limited or restricted)
   a. 4 point scale for GPA can restrict range because there can be missed variability in those who got an “A”
   b. Range is also restricted on the end of SAT scores; those with low SAT scores, probably didn’t go to college (so they are not included in the sample)
4. Skew and relative proportions—if the two variables being correlated have different “skews,” then the correlation between those variables will be reduced
   a. If dichotomous variable, want to ensure groups are equal, otherwise skew will impact
5. Method variance—reduces the correlation (or, smaller than two of the same method)
6. Time—the longer the time between two data points, the smaller the validity correlation
7. Predictions of single events—validity correlations are larger with gathered observations over long periods of time (single events are less predictable than aggregation of events)

Question 20

Interpreting Validity Coefficients (Book)

Answer 20

1. Squared correlations and variance explained (.30 correlation is 9% of variance explained)
2. Estimating practical effects (impact on decision making and predictions)
   a. Binomial effect size display—what percentage (proportion) of the high scorers will perform well on the criterion variable and what percentage of the low scorers will perform poorly
   b. Taylor-Russel tables—tables to inform selection decisions and provide probability of prediction based on an acceptable test score resulting in successful performance
   i. Compare the general success rate to success rate with screening test; is it higher?
   c. Utility Analysis—cost vs benefit; is it worth it? Assign monetary values
   d. Sensitivity/specificity—for detecting categorical differences (ability to produce correct identifications)
   i. Sensitivity—ability to identify those with the disorder
3. TP / (TP + FP)
   ii. Specificity—ability of the test to correctly identify those without the disorder
4. TN / (TN + FP)

Question 21

Aspects of the validation process (Campbell & Fiske (1959) )

Answer 21

1. Validation is typically convergent by independent measurement procedures
2. For new measures, discriminant as well as convergent validation is required
3. Each test is a trait-method unit (variance a result of measurement features and trait content)
4. In order to examine discriminant validity, more than one trait and more than one method must be used

Question 22

MTMMM (Campbell & Fiske (1959) )

Answer 22

1. Heterotrait-monomethod is solid line; heterotrait-heteromethod is broken line *triangle
2. Measures of the same trait should correlate higher than measures of different traits; should also be higher than different traits measured by the same methods