Study Guide 4: Construct Validity

Question 1

Average deviation mean (ADm) index

Answer 1

indicates the degree of disagreement among subject matter experts (SMEs) in the response option selected regardless of whether they, as a group, endorsed an element or not.

Question 2

Construct-irrelevant variance

Answer 2

reproducible variance that is irrelevant to the construct. Two forms: construct irrelevant easiness and construct-irrelevant difficult.

Question 3

Concurrent validity evidence

Answer 3

degree to which test scores are correlated with other relevant variables measured at the same time (e.g. SAT and GPA scores in grade 12).

Question 4

Construct validity

Answer 4

the degree to which test scores can be interpreted as reflecting a particular construct.

Question 5

Content validity or content-related evidence of validity

Answer 5

the degree to which the content of a measure truly reflects the full domain of the construct for which it is used.

Question 6

Content validity index (CVI)

Answer 6

indicates the proportion of subject matter experts (SMEs) that endorse an element as content valid.

Question 7

Convergent evidence

Answer 7

the degree to which test scores are correlated with tests of relative constructs (e.g. self-esteem and happiness).

Question 8

Discriminant evidence

Answer 8

degree to which test scores are uncorrelated with tests of unrelated constructs (e.g. self-esteem and intelligence).

Question 9

Face validity

Answer 9

whether a test appears to measure what it’s supposed to measure, as judged by non-experts. Does not apply to construct validity but carries implications for test use.

Question 10

Internal structure

Answer 10

the way parts of a test are related to each other. Does the actual structure of the test match the theoretical structure of the construct?

Question 11

Modern or unitary view of validity

Answer 11

There is only one type of validity: construct validity. Validity is seen as a property of interpretation and test use. That is, inferences made must be considered in terms of context.

Question 12

Predictive validity evidence

Answer 12

degree to which test scores are correlated with relevant variables measured at FUTURE points in time (e.g. SAT score from grade 12 and GPA score from first year uni).

Question 13

Response processes

Answer 13

The match between the psychological process that respondents actually use when completing a measure and the processes they should use.

Question 14

Samuel Messick

Answer 14

American psychologist who studied construct validity. He focused on inferences made based on test scores and consequences of test use.

Question 15

Standards for Educational and Psychological Testing

Answer 15

A set of testing stanards developed by the American Educational Research Association and APA. It identifies 5 sources of validation evidence: test content, response processes, internal structure, relations to other variables, and consequences of testing.

Question 16

Subject matter experts (SMEs)

Answer 16

Experts in the construct of interest. They are used to obtain both quantitative evidence of content validity and qualitative feedback.

Question 17

Traditional or Trinitian view of validity

Answer 17

Described 3 types of validity (content, criterion-related: concurrent and predictive, and construct: convergent and discriminant). Validity seen as a property of the test. That is, certain tests only require a specific type of validation evidence. Interested in whether test was useful for its intended purpose.

Question 18

Validity

Answer 18

the meaningfulness, usefulness and appropriateness of the inference made from scores obtained on an instrument with a given population, in a given context, for particular use. focused on degree (not all or nothing), use of evidence and theory, interpretation of scores, use of scores.

Question 19

Validation

Answer 19

the process or method by which validity is examined.

Question 20

Susan Embretson

Answer 20

focused on using cognitive modeling to improve upon construct validation. In one approach, she argues of identifying the theoretical mechanisms that underlie item responses, such as information processes, strategies, and knowledge stores. In another approach, she is concerned with the network of relationships of a test score with other variables.

Question 21

Denny Borsboom

Answer 21

argues that tests (NOT INFERENCES) are valid. Also put forward logical arguments such as “a test is valid for measuring an attribute if and only if (a) the attribute exists and (b) variations in the attribute causally produce variations in the outcomes of the measurement procedure.

Question 22

Bruno Zumbo

Answer 22

argued that validity is the explanation for why there is variation in test scores and validation is the process of developing and testing that explanation.

Question 23

Michael Kane

Answer 23

not a validity theorist; best known for trying to provide more explicit guidelines for test validation (contrast to Embretson – he makes it simple)