SCC CH. 3 Construct Validity and External Validity

Question 1

Three requirements of a theory of constructs

Answer 1

1) Operationalize each construct several ways within and across studies.
2) Probing the pattern match between the multivariate characteristic of instances and the characteristics of the target category.
3) Acknowledge legitimate debate about the quality of that match given the socially constructed nature of operations and constructs.

Question 2

14 threats to construct validity

Answer 2

1) Inadequate Explication of Constructs
2) Construct Confounding
3) Mono-Operation Bias
4) Mono-Method Bias
5) Confounding Constructs with Levels of Constructs
6) Treatment Sensitive Factorial Structure
7) Reactive Self-Report Changes
8) Reactivity to the Experimental Situation
9) Experimenter Expectancies
10) Novelty and Disruption Effects
11) Compensatory Equalization
12) Compensatory Rivalry
13) Resentful Demoralization
14) Treatment Diffusion

Question 3

1) Inadequate Explication of Constructs

Answer 3

Failure to adequately explicate constructs may lead to incorrect inferences about the relationship between operation and construct.

Question 4

Construct Confounding

Answer 4

Operations usually involve more than one construct, and failure to describe all the constructs may result in incomplete construct inferences.

Question 5

Mono-Operation Bias

Answer 5

Any one operationalization of a construct both under represents the construct of interest and measures irrelevant constructs, complicating inference.

Question 6

Mono-Method Bias

Answer 6

When all operationalizations use the same method (e.g. self-report), that method is part of the construct actually studied.

Question 7

Confounding Constructs with Levels of Constructs

Answer 7

Inferences about the constructs that best represent study operations may fail to describe the limited levels of construct that were actually studied.

Question 8

Treatment Sensitive Factorial Structure

Answer 8

The structure of a measure may change as a result of treatment, change that may be hidden in the same scoring is always used.

Question 9

Reactive Self-Report Changes

Answer 9

Self-reports can be affected by the participant motivation to be in a treatment condition, motivation that can change after assignment is made.

Question 10

Reactivity to the Experimental Situation

Answer 10

Participant responses reflect not just treatments and measures but also participants’ perceptions of the experimental situation, and those perceptions are part of the treatment construct actually tested

Question 11

Experimenter Expectancies

Answer 11

The experimenter can influence participant responses by conveying expectations about desirable responses, and those expectations are part of the treatment construct as actually tested

Question 12

Novelty and Disruption Effects

Answer 12

Participants may respond unusually well to a novel innovation or unusually poorly to one that disrupts their routine, a response that must then be included as part of the treatment construct description

Question 13

Compensatory Equalization

Answer 13

When treatment provides desirable goods or services, administrators, staff or constituents may provide compensatory goods or services to those not receiving treatment, and this action must then be included as part of the treatment construct description

Question 14

Compensatory Rivalry

Answer 14

Participants not receiving treatment may be motivated to show they can do as well as those receiving treatment, and this compensatory rivalry must then be included as part of the treatment construct description

Question 15

Resentful Demoralization

Answer 15

Participants not receiving a desirable treatment may be so resentful or demoralized that they may respond more negatively than otherwise, and this resentful demoralization must then be included as part of the treatment construct description

Question 16

Treatment Diffusion

Answer 16

Participants may receive services from a condition to which they were not assigned, making construct descriptions of both conditions more difficult.

Question 17

What are 5 targets of generalization?

Answer 17

​- Narrow to Broad
​- Broad to Narrow
​- At a similar level
​- To a similar or different kind
​- Random sample to population members

Question 18

What 2 external validity questions can be asked about whether a causal relationship holds?

Answer 18

1) Over variations in persons, setting, treatments, and outcomes that were in the experiment.
2) For persons, setting, treatments, and outcomes that were not in the experiment.

Question 19

What is secondary analysis?

Answer 19

Reanalyzing data from the experiment to see what happens to the causal relationship as the variable in question is varied.

Question 20

Threats to external validity deal with interaction with…?

Answer 20

o Units
o Treatments
o Outcomes
o Settings

Question 21

What are 5 threats to external validity?

Answer 21

1) Interaction of the Causal Relationship with Units
2) Interaction of the Causal Relationship Over Treatment Variations
3) Interaction of the Causal Relationship with outcomes
4) Interactions of the Causal Relationship with Settings
5) Context-Dependent Mediation

Question 22

Interaction of the Causal Relationship with Units

Answer 22

An effect found with certain kinds of units might not hold if other kinds of units had been studied

Question 23

Interaction of the Causal Relationship Over Treatment Variations

Answer 23

An effect found with one treatment variation might not hold with other variations of that treatment, or when that treatment is combined with other treatments, or when only part of that treatment is used

Question 24

Interaction of the Causal Relationship with outcomes

Answer 24

An effect found on one kind of outcome observation may not hold if other outcome observations were used.

Question 25

Interactions of the Causal Relationship with Settings

Answer 25

An effect found in one kind of setting may not hold if other kinds of settings were to be used

Question 26

Context-Dependent Mediation

Answer 26

An explanatory mediator of causal relationship in one context may not mediate in another context