Study Guide 4: Construct Validity Flashcards
Average deviation mean (ADm) index
indicates the degree of disagreement among subject matter experts (SMEs) in the response option selected regardless of whether they, as a group, endorsed an element or not.
Construct-irrelevant variance
reproducible variance that is irrelevant to the construct. Two forms: construct irrelevant easiness and construct-irrelevant difficult.
Concurrent validity evidence
degree to which test scores are correlated with other relevant variables measured at the same time (e.g. SAT and GPA scores in grade 12).
Construct validity
the degree to which test scores can be interpreted as reflecting a particular construct.
Content validity or content-related evidence of validity
the degree to which the content of a measure truly reflects the full domain of the construct for which it is used.
Content validity index (CVI)
indicates the proportion of subject matter experts (SMEs) that endorse an element as content valid.
Convergent evidence
the degree to which test scores are correlated with tests of relative constructs (e.g. self-esteem and happiness).
Discriminant evidence
degree to which test scores are uncorrelated with tests of unrelated constructs (e.g. self-esteem and intelligence).
Face validity
whether a test appears to measure what it’s supposed to measure, as judged by non-experts. Does not apply to construct validity but carries implications for test use.
Internal structure
the way parts of a test are related to each other. Does the actual structure of the test match the theoretical structure of the construct?
Modern or unitary view of validity
There is only one type of validity: construct validity. Validity is seen as a property of interpretation and test use. That is, inferences made must be considered in terms of context.
Predictive validity evidence
degree to which test scores are correlated with relevant variables measured at FUTURE points in time (e.g. SAT score from grade 12 and GPA score from first year uni).
Response processes
The match between the psychological process that respondents actually use when completing a measure and the processes they should use.
Samuel Messick
American psychologist who studied construct validity. He focused on inferences made based on test scores and consequences of test use.
Standards for Educational and Psychological Testing
A set of testing stanards developed by the American Educational Research Association and APA. It identifies 5 sources of validation evidence: test content, response processes, internal structure, relations to other variables, and consequences of testing.
Subject matter experts (SMEs)
Experts in the construct of interest. They are used to obtain both quantitative evidence of content validity and qualitative feedback.
Traditional or Trinitian view of validity
Described 3 types of validity (content, criterion-related: concurrent and predictive, and construct: convergent and discriminant). Validity seen as a property of the test. That is, certain tests only require a specific type of validation evidence. Interested in whether test was useful for its intended purpose.
Validity
the meaningfulness, usefulness and appropriateness of the inference made from scores obtained on an instrument with a given population, in a given context, for particular use. focused on degree (not all or nothing), use of evidence and theory, interpretation of scores, use of scores.
Validation
the process or method by which validity is examined.
Susan Embretson
focused on using cognitive modeling to improve upon construct validation. In one approach, she argues of identifying the theoretical mechanisms that underlie item responses, such as information processes, strategies, and knowledge stores. In another approach, she is concerned with the network of relationships of a test score with other variables.
Denny Borsboom
argues that tests (NOT INFERENCES) are valid. Also put forward logical arguments such as “a test is valid for measuring an attribute if and only if (a) the attribute exists and (b) variations in the attribute causally produce variations in the outcomes of the measurement procedure.
Bruno Zumbo
argued that validity is the explanation for why there is variation in test scores and validation is the process of developing and testing that explanation.
Michael Kane
not a validity theorist; best known for trying to provide more explicit guidelines for test validation (contrast to Embretson – he makes it simple)
