Study Guide 13: Validity: Consequences and Response Processes Flashcards
Construct
may be conceived of as a concept or a mental representation of shared attributes or characteristics, and it is assumed to exist because it gives rise to observable or measurable phenomena
Intended social and personal consequences:
- Intended consequences: outcomes of the measure/test that developers intend on.
- Social consequences refer to the consequences for society stemming from the use of a measure (e.g., findings might affect theories about depression, funding for mental health programs, etc.)
- This is different from personal consequences which refers to the consequences for the individual stemming from the use of a measure (e.g., findings affect personal diagnoses).
Consequences of testing:
NOT about test misuse
• Refers to the intended consequences and unintended side effects of legitimate test interpretation and use.
• Two aspects of consequential basis: value implications and social consequences
Measurement:
is basically the description of attributes or characteristics in terms of number.
Multilevel construct:
a phenomenon that is potentially differentially meaningful both in use and interpretation at the level of individuals and at one or more levels of aggregation. ** Important to consider the level (individual versus group) at which test scores are being interpreted.
Progressive matrix of validity:
The progressive matrix of validity emphasizes what one needs to consider when validating inferences from our measures (both in terms of interpretation of scores and then use of scores)
Psychometrics
a field of study that focuses on the theory and techniques associated primarily with the measurement of constructs as well as the development, interpretation, and evaluation of tests and measures
Social consequences
this is compared to personal consequence
Unintended social and personal side effects
unanticipated effects on society and the individual stemming from use of a measure. These can be both positive and negative; they are simply not what test developers intended.
What is Cognitive Aspects of Survey Methodology (CASM) research and how does it apply to validation?
CASM is an interdisciplinary movement aimed to: i) improve the quality of self-report data and ii) improve survey applications/design as well as basic cognitive research. It highlights the importance of the substantive aspect (theory and process modeling) of construct validity. Response processes is also listed as one of the five sources of validity in the Standards.
What is a Think-Aloud protocol and what is meant by retrospective and concurrent verbalization? How is verbal probing different?
Think-aloud protocol is a cognitive interviewing technique to evaluate the cognitive process involved in responding to items. How does the participant arrive at their response?
Retrospective verbalization:
complete the items, then go back to item 1 and ask them what they were thinking
Concurrent verbalization:
responding to items and talking your way through the measure
Verbal probing:
respondents are probed for specific information by the interviewer, such as asking the respondent to reformulate an item, or define some key terms in their own words. This is different than Think-Aloud, where the researcher hardly interjects. Verbal probing is often used in combination with think-aloud.
In a single sentence, what was the main overall finding of this study? How did this finding relate to (a) the wording of the items, (b) age/grade level of the respondents, and (c) item content?
The findings indicate that most children had no difficulty in understanding the items, and used mainly two strategies for responding to the items: (1) an absolute strategy, in which children used absolute statements to indicate the presence or absence of something that is important for their judgment of their satisfaction with life, and (2) a relative strategy, in which children used relative or comparative statements.
