Validity

Question 1

Construct Validity

Answer 1

“an overall evaluation of the degree to which empirical evidence and theoretical rationales support the adequacy and appropriateness of interpretations and actions on the basis of test scores” (Messick, 1995, p.741)

Does the data that you obtain from a questionnaire align with theory and evidence?

What is the meaning of test scores?

Can we trust the data we obtain?

Question 2

Major sources of invalidity

Answer 2

Under Representation: test is too narrow, fails to include important components of a construct

Irrelevance: Test is too broad - contains components that are not actually tapping into the focal construct

Difficulty: Test is too difficult or too easy

Question 3

Six Aspects of Validity

Answer 3

1. Content
2. Substantive
3. Structural
4. External
5. Consequential
6. Generalizability

Question 4

Content Validity

Answer 4

What should the questionnaire include?

Relevance, representativeness, and technical quality of measure

Items that negatively impact validity:

 - Unclear, complicated, overly colloquial
 - No variability in responses
 - "Double-barrelled"

Question 5

Substantive validity

Answer 5

How do respondents interpret/make sense of the questionnaire and its items?

Is there alignment between what’s supposed to be measure and participants’ interpretations?

E.g., “Members of this team are on the same page”

Towards understanding how the participants themselves make sense of the measure

Question 6

Structural Validity

Answer 6

Does the data fit/align with the theoretical propositions of that construct?

Data = Theory/Evidence?

The main question here is does data derived from questionnaire align withtheoretical propositions/model?

If there is a high degree of alignment, this provides evidence of structural validity

Question 7

External Validity

Answer 7

How does the data relate to data from other measures of other relevant structures?

Data = Theory/Evidence

Question 8

Consequential validity

Answer 8

What are the implications that may result from using the questionnaire?

1) Can the measure be used as a ‘basis for action’?
2) Are there any potential consequences of test use?

Examples:

Can a depression questionnaire be used to guide treatment?

Are there any potential negative repercussions from completing a germ awareness questionnaire?

Question 9

Generalizability validity

Answer 9

To what extent do results generalize across populations, contexts, and tasks?

Participants from samples of various populations (e.g., different age groups, contexts, demographics) complete your measure (and other measure)

Do data obtained in these studies align with the data you originally obtained? Why/why not?

Examples

Are the consequences of administering the Germ Awareness Questionnaire different for children v adults?