Lecture 9: Validity

Question 1

Reliability

Answer 1

degree to which differences in test scores reflect differences in psychological attribute that affects test scores (does the test’s score reflect something with precision?

Question 2

Validity

Answer 2

“the degree to which evidence and theory support the interpretations of test scores for proposed uses”

Question 3

3 Points about validity

Answer 3

1. Validity is about the interpretation of test scores in terms of a specific psychological construct; it is not about the test itself 2. Validity is matter of degree; it is not all or none 3. Validity is based on solid empirical evidence and theory (use information from research, not just choice)

Question 4

What is the importance of theory for validity

Answer 4

questionnaires and test require a theory that defines what the construct is; otherwise it’s not possible to understand what the test scores mean (if you can’t define it, you don’t know if you’re measuring it)

Question 5

Why is it important to measure validity? (what could be detrimental if you don’t)

Answer 5

if the psychological meaning of test scores is misinterpreted, then 1. Psychological conclusions based on those scores may be mistaken 2. Decisions made about people (based even partly on those scores) may be misguided and potentially unfair or dangerous

Question 6

What are the dimensions of construct validity?

Answer 6

1. Test Content (what are we measuring) 2. Internal Structure (Factor analysis; what are the dimensions? How do they relate) 3. Response processes (how someone thinks about an item, what are they thinking about, how does that relate to what we’re trying to ask) 4. Consequences of use (the impact—what we want or what we don’t) 5. Associations with other variables (how we’ve been thinking about it in the past; empirical evidence)

Question 7

What are the three types of validity? (from book reading)

Answer 7

1. Content
2. Criterion—degree to which test scores can predict specific criterion; de-emphasizes conceptual meaning or interpretation of test scores
   a. Concurrent and predictive validity (criterion is now thought to be under the construct)
3. Construct (now more focused)
   a. Test content—content of the test match what should be on the test (expert rating)
   i. Construct-irrelevant content and construct underrepresentation
   ii. Face validity is close to content (but looks at the common people)
   b. Internal structure—does the internal structure match what it should be (factor analysis)
   c. Response processes—do the psychological processes that respondents use when completing a measure match what they should (interviewing and eye tracking)
   d. Associations with other variables—do the associations with other variables match what they should be (convergent, discriminant, concurrent, and predictive correlations)
   e. Consequences of use—actual consequences of using a measure match the consequences that should be seen (evaluation of consequences, differential impact and systematic changes [MCAT content reflected how premeds were taught])

Question 8

What are the three types of validity from Cronbach and Meehl?

Answer 8

1. Criterion
   a. Predictive—criterion obtained after test is given
   b. Concurrent—test score and criterion obtained at the same time
2. Content—showing the test items area sample of a universe in which the investigator is interested; deductive
   a. Acceptance of the universe of the content as defining the variable to be measured is essential
3. Construct—when a test is interpreted as a measure of some attribute or quality which is not “operationally defined”

Question 9

What is content validity?

Answer 9

“the systematic examination of the test content to determine whether it covers a representative sample of the behavior domain to be measured”; the relationship between the content of a test and some well-defined domain of knowledge or behavior (relationship between what it is on the test and what you think should be on the test based on the theory); good match between content and domain=high content validity); does the actual content of a test match the content that should be included on the test?

Question 10

Steps for content validity

Answer 10

1. Reflect the full scope of the construct (vs construct under-representation) 2. Systematically exclude anything besides the construct (vs construct irreverent variance); evidence via expert ratings

Question 11

Construct underrepresentation

Answer 11

The degree to which a test fails to capture important aspects of the given construct

Question 12

Irrelevant variance

Answer 12

the degree to which test scores are affected by processes extraneous to the intended construct

Question 13

Steps to assess content validity

Answer 13

1. Describe the content domain (in terms of boundaries/limits and structure) 2. Determine the areas of the content domain that are measured by each item (go item by item, what am I measuring?) 3. Compare the structure of the test with the structure of the content domain (do you want two domains even?)

Question 14

Concerns with content validity

Answer 14

short forms of a test (ex. Shortened depression screening tool—it only has two items, maybe missing people or over diagnosing); content overlap (ex. Agitation or tension; if you’re measuring depression, but including things like tense or agitation, it’s more anxiety; overlap between two different questionnaires of two different constructs)

Question 15

Final thoughts on content validity

Answer 15

content validation is a process rather than a statistical analysis (really a process where you’re going through each item and domain); content validity differs from internal consistency reliability (in internal consistency we want the items to hang together and mean the same thing; content validity is similar but more related to the content of the construct, not just if the items measure the same thing)

Question 16

Face validity

Answer 16

if a test looks like it measures its target construct (“how sad are you?” For depression); don’t confuse this with the empirical approach (does not mean it actually measures what you’re interested in); concern: malingering (subject to response bias—all questions are, but more subject; low face validity might be better for malingering)

Question 17

Internal Structure (structural validity)

Answer 17

does the actual internal structure of a test match the structure that the test should possess; dimensionality and factor analysis (if a test has different dimensions, how it is measuring these and how do they relate to one another)

Question 18

Internal structure validity issues evaluated through factor analysis

Answer 18

1. Number of factors 2. Meaning of factors 3. Associations between factors (if more than one factor) 4. Order of factors (do they relate to the construct overall or are they separate? Top: Depression, Second Level: 1. Cognition 2. Somatic, Third level: what relates to those)

Question 19

Reponses Process Validity

Answer 19

do the psychological processes that actually shape peoples’ response to the test match the processes that should shape their response? (understand what the question means to the individual taking the test; people have different processes; interviewing after a questionnaire helps get a general sense of individual’s response)

Question 20

Two types of evidence for response process validity

Answer 20

Evidence: 1. Direct evidence: interviews with respondents, “think alouds” 2. Indirect evidence: eye-tracking, response times, statistical analysis, and experimental studies of process (use if you can’t give interview or if you think they’re lying)

Question 21

Consequences of Testing Validity

Answer 21

do the actual consequences of using a test match the consequences that should be seen?; Controversial as a facet of validity

Question 22

Evidence for consequences of testing validity

Answer 22

evidence: 1. Evidence of intended effects 2. Evidence regarding unintended differential impact on groups (unbiased test could have biased results; women might have higher emotional intelligence than men, but if using for a screening for grad school, you’ll choose more women) 3. Evidence regarding unintended systemic effects (choosing more women for your program over time, more woman professors and faculty members, long-term effects on school and even profession)

Question 23

Associations with other measures validity

Answer 23

does the actual associations with other measures match the associations that it should have with those measures? (are the scores obtained a good measure of the construct; extent to which the scores obtained on measure are consistent with the construct and measures related to this construct);

Question 24

Evidence for associations with other measures validity

Answer 24

); evidence: 1. Convergent Validity (concurrent/predictive validity; expect to find significant correlations between measures of behaviors related to our construct and our test; different measures of construct should be correlated also) 2. Discriminant (divergent) validity (behaviors not associated with our construct of interest should be uncorrelated with our test)

Question 25

How do you evaluate association with other measures correlations?

Answer 25

Some correlations .45 could be a low level of convergent OR divergent, to determine it you look at what you expect (nomological network)

Question 26

Association with other measure key questions 1. How do you know which scales to measure or examine? 2. How do you know what pattern of convergent or discriminant associations to expect?

Answer 26

Constructs Nomological Network

Question 27

Constructs Nomological Network

Answer 27

network of associated constructs, behaviors, and properties

Question 28

Assessing criterion-related validity

Answer 28

(a criterion is an outcome that you want to be associated with your measure—what you hope your measure will predict) correlation of test scores with the outcomes of decisions that are made 1. Predictive validation strategies (the “ideal” approach; 1. Obtain test scores 2. Obtain performance measures and correlate these with test scores; gives you a validity coefficeint) 2. Concurrent validation strategies (a practical alternative; test scores and criterion scores from a preselected population are obtained at the same time)

Question 29

Predictive validation strategies

Answer 29

the “ideal” approach; 1. Obtain test scores 2. Obtain performance measures and correlate these with test scores; gives you a validity coefficeint

Question 30

Concurrent validation strategies

Answer 30

a practical alternative; test scores and criterion scores from a preselected population are obtained at the same time

Question 31

Advantages to concurrent validation

Answer 31

1. Much more practical than predative validation 2. Easier than predictive validation 3. Concurrent validity coefficients are often similar to those of predative validation

Question 32

Statistical and conceptual problems with concurrent validation

Answer 32

quality of the predictor (often there are many factors that relate to the criterion); range restriction (reduces correlation between predictor [test score] and criterion [outcome measure]; if you’re looking at GRE score and success in grad school, you would miss those who didn’t get into grad school; can be direct, through selection by the predictor; can be indirect, through decisions based on the criterion); goal is often to screen out failures (allowed for in predictive validation not concurrent validation; predictive is much better than screening tool)

Question 33

Incremental Validity

Answer 33

a test much demonstrate that it has better predictive ability than data from an assessment; demonstrate validity beyond existing assessments; concern: who decides the Gold Standard; supposed to choose gold standard, but some don’t just so they can prove this

Question 34

Interpreting validity coefficients

Answer 34

correlation coefficients range from 0-1; don’t expect huge validity coefficients (usually not larger than 0.5); squaring the validity coefficient tells about the amount of variance in the criterion that can be explained by the predictor (ex. Using a test of cognitive ability to predict results in a validity coefficient of 0.5. We can say 25% of the variance in job performance is accounted for by cognitive ability)

Question 35

Methods of Construct Validation

Answer 35

1. Correlational study (correlations between measures of certain behaviors [that are either related to or unrelated to our construct of interest] and our test; meta-analysis: gather many validity studies and combining them) 2. Factor analysis (analyzing which groups of items “hang” together) 3. Experimental manipulation (manipulate the construct of interest [i.e., induce fear] and see if it relates to different scores on our test) 4. Multi-trait multi-method matrix (MTMM)

Question 36

Multitrait-Multimethod Matrix (MTMM):

Answer 36

based on convergent and discriminant validity; three elements 1. Multitrait—measures different traits 2. Multimethod—uses various methods to measure those traits 3. Matrix—a table comprised of correlations between methods and traits measures

Question 37

Base rate

Answer 37

level of performance on criterion in the general population; e.g., if 75% of the population successful, the base rate is 0.75

Question 38

Selection ratio

Answer 38

ratio of positions to applicants; e.g., if 30 people apply for 3 jobs, the selection ratio is 10% or 0.10

Question 39

True Positive (TP)

Answer 39

when a test predicts success and the person is actually success

Question 40

True Negative (TN)

Answer 40

when a test predicts failure and a person actually fails

Question 41

False Positive (FP):

Answer 41

When a test predicts success and the person actually fails

Question 42

False Negative (FN):

Answer 42

when a test predicts failure and a person actually succeeds

Question 43

Effect of base rate on decisions: when the base rate is large

Answer 43

most everyone would be successful, but there are a limited # of positions, so there will be a high number of true positives and a high number of false negatives

Question 44

Effect of base rate on decisions: when base rate is small

Answer 44

hardly anyone would be successful, but must fill a set # of positions, so there will be a high number of true negatives and a high number of false positives

Question 45

when are tests used as predictors are most likely to have an impact on accurate decision making

Answer 45

when the base rate is moderate (0.5)

Question 46

Effect of selection ratio on decisions

Answer 46

: if selection ratio is high (# of positions and # of applicants almost equal), doesn’t really matter what method of prediction you use because you’re taking almost everyone who applies; validity has the biggest impact on correct decision making when selection ratio is low

Question 47

sensitivity

Answer 47

the probability of a test to correctly identify individuals who have the disorder; sensitivity= true positives/ (true positives and false negatives; i.e. everyone who actually has the disorder);

Question 48

specificity

Answer 48

the probability of a test to correctly identify individuals who do NOT have the disorder; specificity=true negatives/(true negatives + false positives)

Question 49

Internal validity

Answer 49

: it depends on the strength or soundness of the design and analysis of the study in which a scientific question is evaluated, including any study that validates a psychometric test or question

Question 50

History

Answer 50

(Events, other than the experimental treatments or the true effect among the variables, that have influenced results),

Question 51

Maturation

Answer 51

(During the study, some kind of psychological change occurs within people completing the test that affects how test scores are related to scores on other tests and questionnaires)

Question 52

Testing

Answer 52

Exposure to some known or unknown pretest or intervening assessment influences performance on your test)

Question 53

Instrumentation

Answer 53

(Testing instruments or conditions in which people complete the tests are inconsistent; or the pre- and post-tests are not equivalent in some manner),

Question 54

Statistical Regression

Answer 54

Scores of subjects that are very high or very low tend to regress towards the mean during retesting),

Question 55

Selection

Answer 55

Systematic differences exist in subjects’ characteristics between treatment or comparison groups which affect the performance of the test)

Question 56

Mortality

Answer 56

(Attrition affects the representativeness of the group you are studying and can lead to an over-estimate, under-estimate or an unreliable result),

Question 57

Diffusion of Treatments (

Answer 57

People in one group are able to communicate with the other group and affect the manner in which the other group completes the questionnaire),

Question 58

Effects of Pretesting/Order Effects

Answer 58

This is when one test in your battery of tests affects how people complete a subsequent test. Completing one test may lead to an improvement, increase in skill

Question 59

Placebo Effects or Expectancy Effects

Answer 59

This is when you get an effect because the participant expects something to happen even those there no “active” ingredient or agent)

Question 60

Demand Characteristics (

Answer 60

This is when people completing the questionnaires respond in a manner that they believe is expected of them

Question 61

Experimenter Bias

Answer 61

The tests developers’ ideas about how the construct should be defined and operationalized is in some manner “biased” and as a result, out of line with how most people would define and measure it, leading to the creation of a test that produces very different results