Lecture 1 - Substantive Validation

Question 1

What is the purpose of substantive validation?

Answer 1

To determine if the items generated are conceptually linked to the construct of interest.

Question 2

What are the 4 steps in substantive validation?

Answer 2

1) Identify a small group of Subject Matter Experts (SME) to rate each item in terms of:
Relevance / representativeness to the construct
Clarity

2) Administer questionnaire comprising the definition of your construct (and its respective dimensions).

3) Data Analysis (performed on representativeness data):
Compute inter-judge agreement across all items (using Perreault & Leigh’s
(1989) variation of Cohen’s kappa)
Compute inter-judge agreement for each item (Content Validity Ratio)

4) Retention criterion:
Items in which all judges rated an item positively (i.e., CVR approaches I) will be retained.

Question 3

What is the inter-rater agreement across all items (IR)?

Answer 3

The inter-rater agreement across all items (IR) reflects the degree to which experts agree across all the items. It does not inform you about the quality of any one item.

Question 4

What is the content validity ratio (CVR)?

Answer 4

The content validity ratio (CVR) is a measure of the extent to which experts endorse an item.