Chapter 5-validity

Question 1

Validity def

Answer 1

The extent to which a test measures the attribute/construct it is designed to measure
Does the test measure what it was designed to measure?
-> Not a yes or no question - question of DEGREE
-> One of the most important characteristics of the test.

Question 2

Guidelines regarding validity (3)

Answer 2

(1) Do NOT accept a test’s name as an indicator of what the test measures.
(2) Validity is NOT a yes/no decision
(3) Validity evidence tells how well the test measures what it is intended to measure.
-> Diff types of evidence can be generated for diff types of validity

Question 3

What do we mean when we say that “Validity is NOT a yes/no decision”

Answer 3

• It comes in degrees and applies to a particular USE and a particular POPULATION
• It is a process: An ongoing, dynamic effort to accumulate evidence for a sound scientific basis for proposed test score interpretations

Question 4

3 Types of Validity

Answer 4

Content, Criterion, Construct

Question 5

Subtypes of Criterion validity

Answer 5

Concurrent, Predictive

Question 6

Subtypes of Construct validity

Answer 6

Convergent, Divergent

Question 7

Face Validity

Answer 7

Whether a test appears to measure what it is supposed to measure (does it appear valid).
Mere appearance that a measure has validity.
=> Not sufficient evidence of validity

Question 8

A test with high face validity may: (3)

Answer 8

(1) Induce cooperation and positive motivation before and during test administration
(2) Reduce dissatisfaction and feelings of injustice among low scorers
(3) Convince policymakers, employers, and administrators to implement the test

Question 9

There are situations where test designers make ON PURPOSE a test with low face validity. Why?

Answer 9

Sometimes a test with low face validity can elicit more honest responses

Question 10

Content Validity def

Answer 10

Degree to which ELEMENTS OF A TEST are representative of the domain/construct of interest.
-> Evaluate how adequately the test samples the domain or content of the construct.
-> More QUALITATIVE than Quantitative

Question 11

Establishing content validity (3)

Answer 11

(1) Describe the content domain: Identify the boundaries of the content domain + Determine the structure of the content domain.
(2) Inspect test - Expert judgment
(3) Form judgment that the test measures what it is supposed to measure… without gathering any external evidence
+ Content of the items must be carefully evaluated (wording appropriateness…).

Question 12

When is content validity high?

Answer 12

When test content is a representative sample of the tasks that define the content domain
+
When the items do not measure something else

Question 13

However, content validity is not enough to determine that the test is valid. Why?

Answer 13

No information about relation of test to external constructs or external variables

Question 14

Criterion Validity def

Answer 14

Effectiveness of the test in predicting narrowly/specifically identified variables that are thought to be DIRECT measures of the construct.
-> How well a test corresponds with a particular criterion.

Question 15

Criterion def

Answer 15

A standard that researchers use to measure outcomes such as performance, attitude, performance.
-> Standard against which the test is compared

Question 16

Objective criterion characteristics (2)

Answer 16

Observable and Measurable
E.g., Number of accidents, days of absence

Question 17

Types of criterion

Answer 17

Objective & Subjective criterion

Question 18

Subjective criterion

Answer 18

Based on a person’s judgement
E.g., Supervisor ratings, peer ratings

Question 19

Concurrent Validity

Answer 19

Comes from assessments of the simultaneous relationship between the test and the criterion.
Criterion available at THE same time as test
-> Can also be used when a person does NOT know how they will respond to the criterion measure.

Question 20

Predictive validity

Answer 20

The forecasting function of tests.
Degree to which test scores accurately predict scores on a criterion measure.
-> Criterion measure available in the future

Question 21

What happens if the criterion measures FEWER dimensions than those measured by the test?

Answer 21

This decreases the evidence of validity based on its content because it has underrepresented some important characteristics
-> Underrepresentative

Question 22

Criterion contamination def

Answer 22

If the criterion measures MORE dimensions than those measured by the test

Question 23

Validity coefficient def

Answer 23

Relationship between a test and a criterion.
-> Tells the extent to which the test is valid for making statements about the criterion.

Question 24

What’s the range of validity coefficients?

Answer 24

Validity coefficients: Correlation between test and criterion
-> Rarely greater than r = .60
-> If higher than that → alternative test

Question 25

Comparison of validity coefficient in psychology and medicine

Answer 25

Psychological tests can provide information that is AS VALID as common medical tests

Question 26

Factors Limiting Validity Coefficients (3)

Answer 26

(1) Range of Scores
(2) Unreliability of Test Scores
(3) Unreliability in Criterion

Question 27

[Factors Limiting Validity Coefficients] Explain (1) Range of Scores

Answer 27

Restricted range of scores decreases validity coefficients because it diminishes the Test score & criterion score correlation

Question 28

Explain (2) Unreliability of Test Scores and how we deal with it

Answer 28

Low reliability decreases validity coefficients.
Solution: Correction for attenuation - validity coefficient if we had perfect realibility of test scores

Question 29

Explain (3) Unreliability in Criterion and how we deal with it

Answer 29

Low reliability decreases validity coefficients
Solution: Correction for attenuation - Correcting for unreliability in test (predictor) & criterion

Question 30

What’s the correction for attenuation - Validity coeff

Answer 30

Observed validity coeff / sqrt(reliability coefficients)
-> See what would the validity coeff be if test and/or criterion were more reliable

Question 31

Take aways of Criterion Validity (2)

Answer 31

(1) Importance of choosing appropriate criterion
(2) Small validity coeff can have practical utility

Question 32

Construct Validity def

Answer 32

“All encompassing” type of validity. Process of Construct validation
-> In construct validity evidence, no single variable can serve as the criterion.
-> Extent to which your test or measure accurately assesses what it’s supposed to

Question 33

Construct validation def

Answer 33

Assembling evidence about what a test means. Done by showing the relationship between a test and other tests and measures.
-> Over a series of studies, the meaning of the test gradually begins to take shape.

Question 34

‘Limitations’ of criterion validity

Answer 34

For some constructs it’s easier, other it’s difficult to find a criterion.
-> E.g. criterion for trust, chronic pain (…)? Hard to identify appropriate criterion.
-> Thus, other ways of establishing validity of test scores → construct validity

Question 35

Psychological Constructs def (4)

Answer 35

(1) Abstract concepts used to refer to psychological attributes (e.g. intelligence, love, beauty)
(2) Exist in theory; generally LATENT; not directly observable (EXCEPTIONS: e.g. reaction time, absenteeism)
(3) Important in describing or understanding human behavior -> Its existence EXPLAINS why/how something is happening
(4) Can observe and measure the behaviors that show evidence of these constructs

Question 36

Construct Explication

Answer 36

How a particular construct is manifested and how such manifestations can be measured.

Question 37

How to gather Evidence of Construct Validity (2)

Answer 37

(1) Gathering Theoretical evidence
(2) Gather Psychometric evidence

Question 38

Explain how we gather THEORETICAL evidence of construct validity (2)

Answer 38

(1) Establish nomological network - identifying all possible relationships
(2) Based on this theoretical work → Propose experimental hypotheses
-> If what we think is true, what would be the evidence to support this relationship

Question 39

Nomological Network (3)

Answer 39

Consists of:
(1) Constructs (e.g. job satisfaction)
(2) Their observable manifestations (e.g. smiles, productivity, positive feedback)
(3) The relations within and between constructs and their observable manifestations (e.g. positive feedback related to productivity)

Question 40

Explain how we gather PSYCHOMETRIC evidence (6)

Answer 40

(1) Content validity
(2) Criterion validity
(3) Reliability of the test
(4) Experimental interventions
(5) Convergent evidence of validity
(6) Discriminant evidence of validity

Question 41

[Gathering psychometric evidence] Evidence of validity based on content (2)

Answer 41

(1) No construct underrepresentation: Does the test sample adequately from the construct domain?
(2) No irrelevant construct representation: Does the test properly exclude content that is unrelated to the construct?

Question 42

[Gathering psychometric evidence] Evidence of validity based on relations with criteria

Answer 42

Are the relations of the test with the external criteria as would expected based on theory?

Question 43

[Gathering psychometric evidence] Evidence of validity based on reliability of the test

Answer 43

E.g. test-retest/internal consistency not too low or too high given the construct

Question 44

[Gathering psychometric evidence] Evidence of validity based on experimental interventions

Answer 44

Provide evidence of situational changes that should influence test scores based on theory
-> E.g. education influencing scores on an achievement test
-> Medication influencing scores on an anxiety test

Question 45

[Gatheting psychometric evidence] Convergent Validity (2)

Answer 45

Extent to which two measures that are supposed to be related are actually correlated
When a test scores correlate with:
(1) Other measures of the SAME construct, or
(2) Measures of constructs to which the test should be related based on theory (think nomologic net)

Question 46

[Gatheting psychometric evidence] Discriminant (Divergent) Validity

Answer 46

Test scores are uncorrelated with:
Measures of constructs to which the construct should NOT be related based on theory (think nomologic net)

Question 47

Problems with Content validity (3)

Answer 47

(1) Educational setting: content validity has been of greatest concern in educational testing (score on this test represent comprehension of subject) BUT many factors can limit performance on test
(2) Unclear boundaries: hard to separate types of validities
-> It’s often hard to separate “content coverage” (content validity) from whether the test actually measures the underlying concept (construct validity), leading to blurred boundaries.
(3) Doesn’t consider the relationship of contruct w external variables/constructs

Question 48

[Problems with Content validity] How can we answer unclear boundaries: hard to separate types of validities?

Answer 48

Content validity evidence offers some unique features. Logical rather than statistical.

Question 49

Construct underrepresentation

Answer 49

CONTENT validity. Failure to capture important components of a construct.

Question 50

Construct-irrelevant variance

Answer 50

CONTENT validity. Occurs when scores are influenced by factors irrelevant to the construct.
-> E.g., a test of intelligence might be influenced by reading comprehension, test anxiety, or illness.

Question 51

Would a validity coeff of .40 always be considered good?

Answer 51

NO. All validity coefficient don’t have the same meaning.

Question 52

Several issues of concern when interpreting validity coefficients (9)

Answer 52

(1) All validity coefficient don’t have the same meaning
(2) The conditions of a validity study are never exactly reproduced. E.g. If you take the GRE to gain admission to graduate school, the conditions under which you take the test may not be exactly the same as those in the studies that established the validity of the GRE.
(3) Criterion-related validity studies mean nothing UNLESS the criterion is valid and reliable.
(4) Validity study might have been done on a population that does not represent the group to which inferences will be made.
(5) Be sure the sample size was adequate
(6) Never Confuse the Criterion with the Predictor (GRE & success in grad school example)
(7) Check for Restricted Range on Both Predictor and Criterion: Correlation requires that there be variability in both the predictor and the criterion.
(8) Review Evidence for Validity Generalization (may not be generalized to other similar situations)
(9) Consider Differential Prediction: Predictive relationships may not be the same for all demographic groups.

Question 53

Differential Prediction

Answer 53

Predictive relationships may NOT be the same for all demographic groups.
-> The validity for men could differ in some circumstances from the validity for women.
-> Under these circumstances, separate validity studies for different groups may be necessary.

Question 54

Criterion-Referenced Tests

Answer 54

Have items that are designed to match certain specific instructional objectives. Designed to measure student performance against a fixed set of predetermined criteria.

Question 55

Validity studies for the criterion-referenced tests

Answer 55

Would compare scores on the test to scores on other measures that are believed to be related to the test.

Question 56

Validity & Reliability relationship

Answer 56

A measurement procedure MUST BE RELIABLE (consistent) in order to be VALID.
-> A measurement procedure can be reliable, but not necessarily valid.