Chapter 2: Internal and External Validity

Question 1

What does “plausible rival hypothesis” refer to?

Answer 1

Those competing interpretations that might be posed to explain the findings of a particular study.

Question 2

What are the four types of validity?

Answer 2

1. Internal
2. External
3. Construct
4. Data-evaluation

Question 3

What is methodology?

Answer 3

A way of thinking that relates directly to how one thinks about a study.

Question 4

What is a crucial part of designing an experiment?

Answer 4

Identifying its purpose and specific questions at the outset.
- This helps prioritize types of validity.

Question 5

What is internal validity?

Answer 5

The extent to which the intervention rather than extraneous influences can be considered to account for the results, changes, or differences among conditions.
- The results can be attributed with little or no ambiguity to the effects of the IV.

Question 6

What are threats to internal validity?

Answer 6

Factors/influences other than the IV that could explain the results.
- These are the problems that could emerge in the design or execution of the study.

Question 7

What are the major threats to internal validity?

Answer 7

1. History
2. Maturation
3. Testing
4. Instrumentation
5. Statistical regression
6. Selection biases
7. Attrition
8. Diffusion of treatment

Question 8

What is history?

Answer 8

Any event, other than the IV, occurring in the experiment or outside the experiment that may account for the results.
- These are events common to all subjects in everyday life.
- The influence of such historical events might alter performance and can be mistaken for an effect resulting from the experimental manipulation or intervention.

Question 9

What does history not refer to?

Answer 9

Events that happen to all of us all of the time. Our individuals histories and experiences.
- Rather, history refers to any event that happens to virtually all of the subjects that might explain the findings of an experiment (e.g. COVID-19).

Question 10

Can history occur during the experiment?

Answer 10

Yes, it includes events that take place during the experiment as well.
- When subjects are run in a group, unplanned events (e.g. fire drill, medical emergency of one participant) may disrupt administration of the intervention and reduce/enhance the influence performance of the participants.

Question 11

What is needed for history to be considered a threat for a given experiment?

Answer 11

1. A historical event occurred when the experiment was run.
2. This even is a plausible interpretation of the findings.

Question 12

What is maturation?

Answer 12

Processes within the participants that change over time and includes growing older, stronger, wiser, and more tired or bored.
- Can arise when there is more than one assessment occasion.

Question 13

When is maturation a problem?

Answer 13

If the design cannot separate the effects of maturational changes from the intervention.
- That is, are changes within the individual over time a plausible explanation of the findings?
- The time frame of the study is relevant to invoking this threat. In general, maturation has greater impact over time.

Question 14

What is testing?

Answer 14

The effects that taking a test one time may have on subsequent performance on the test.

Question 15

Does repeated testing always lead to improvement?

Answer 15

No, and if it does, there are obvious limits so that endlessly retaking some measure will not lead to continued improvement.

Question 16

How can you account for testing effects?

Answer 16

By using a control condition with repeated testing but without the special experimental manipulation or intervention.

Question 17

Which threats to internal validity often go together?

Answer 17

History, maturation, and testing.

Question 18

What is the threat of instrumentation?

Answer 18

Changes in the measuring instrument or measurement procedures over time.

Question 19

What is the most common situation where instrumentation is a threat to internal validity?

Answer 19

When ratings are made by judges or oneself and somehow the standards or criteria for making those ratings change over time.
- Changes in the DV over the course of a study may result from changes in scoring criteria, rather than changes in what is being rated.

Question 20

How can instrumentation affect substantive conclusions about changes over time in clinically and socially relevant domains?

Answer 20

Think about the case of ASD: at one point in history, being diagnosed with ASD was a rare occurrence. However, now diagnoses of ASD are more common.
- Instrumentation is involved: The ways in which cases are being identified (referred to as ascertainment) are much more thorough and comprehensive than it has been before. In addition, the definition of what counts as the disorder has changed.

Question 21

How can the conditions or context of administration influence the nature of a measure, even when holding all of the items in the measure constant?

Answer 21

E.g. a study in China: the first time the participants filled in a survey, the measure was not part of the study. However, the second time they filled in the survey, it was part of a study and they had to sign an informed consent. Thus, the context changed.

Question 22

How can the instrument change and be the same at the same time?

Answer 22

It is possible that the items remain the same, but the items may have a different meaning because of the social context of a given point in time.

Question 23

What is response shift?

Answer 23

Changes in a person’s internal standards of measurement.
- Includes the case of judges who rate athletic performance or prisoners up for parole.
- Phenomenon can be more general any time there might be a change in values, perspective, or criteria that lead to evaluation of the same or similar situations, behaviors, and states, in a different way.

Question 24

What is the hello-goodbye effect?

Answer 24

Before treatment, psychotherapy clients might see their lives as especially bleak or perhaps even distorted a little. At the end of treatment, they respond now by having altered their threshold for noting symptoms or seeing their lives differently even though the symptoms may not have changed.

Question 25

What does response shift reflect?

Answer 25

A change in the threshold for answering a particular way. - The threshold may be influenced by historical and maturational changes in the individual or context in which the instrument is embedded.

Question 26

When is instrumentation a threat to internal validity?

Answer 26

In any instance in which differences might be attributed to a change in the instrument or to a change in the criteria used to complete that instrument.

Question 27

What is the difference between testing and instrumentation?

Answer 27

Testing refers to changes in the individual over time (due to experience and practice on the measure), while instrumentation is about changes in the instrument device or how the measure is used. - Note that both can be prevented by having a control group.

Question 28

What is statistical regression?

Answer 28

The tendency for extreme scores on any measure to revert (or regress) toward the mean of a distribution when the measurement device is re-administered.

Question 29

What are three ways to help protect against statistical regression?

Answer 29

1. Assign participants randomly to an experimental or control condition.

Question 29

What are three ways to help protect against statistical regression?

Answer 29

1. Assign participants randomly to an experimental or control condition. - That way, regression, if present, will affect all groups, and one can see if the experimental manipulation led to changes beyond what was evident in the control group. 2. Use measures that are known to have high reliability and validity. - Regression is a function of error of the measure. The greater the error, the more likely there will be regression. 3. Test everyone twice for the pretest and select only those individuals who were extreme on both occasions. - These people are likely to have the characteristic of interest. - Often not feasible, which is why it rarely occurs.

Question 30

What are selection biases?

Answer 30

Systematic differences between groups before any experimental manipulation or intervention is presented.

Question 31

How are selection biases controlled for?

Answer 31

By randomly assigning subjects to conditions so that so that any subject characteristics that may introduce a selection bias are dispersed among groups roughly equally or at least unsystematically.

Question 32

Where does selection bias often arise?

Answer 32

In clinical, counseling, and educational research where intact groups are selected (e.g. patients from separate clinics or hospital). - In prevention programs, comparisons are often made among classes, schools, or school districts that do or do not receive an intervention.

Question 33

When is selection bias not a threat to internal validity?

Answer 33

When you begin with a select group because the purpose of the study is to identify different groups and elaborate their unique characteristics.

Question 34

What is attrition?

Answer 34

Loss of subjects, which may serve as a threat to internal validity.

Question 35

When does attrition occur?

Answer 35

When an investigation spans more than one session.

Question 36

Why is attrition a problem?

Answer 36

The groups of participants are no longer randomly comprised, and subjects, on the basis of variables we do not really know, elected to pull themselves out of the group. - This could cause us to have difficulty in detecting selection bias, given that the groups are no longer randomly comprised.

Question 37

How can attrition affect overall group performance?

Answer 37

Group performance on the measures may be due to loss of subjects who scored in a particular direction, rather than to the impact of an experimental manipulation or intervention.

Question 38

Where does diffusion/imitation of treatment occur?

Answer 38

In any experiment where groups are exposed to different procedures. - More likely to be a problem in intervention research. In this case, it is possible that the intervention given to one group may be provided accidentally to all or some subjects in a control group as well. This dilutes the effects of the treatment and alters what the investigator concludes about the efficacy of treatment.

Question 39

How does diffusion threaten internal validity?

Answer 39

It equalizes performance of treatment and control groups and thus reduces/distorts effects of the intervention obtained in the study.

Question 40

What happens when those in the intervention condition do not receive the intervention?

Answer 40

A diffusion of the no-treatment condition occurs. - This once again means that the conclusions at the end of the treatment will be misleading.

Question 41

What is a general rule to prevent/limit diffusion?

Answer 41

Ensuring that individuals assigned to a particular condition received that condition and only that condition.

Question 42

What is the threat of special treatment or reaction of controls?

Answer 42

A special circumstance in which an intervention program is evaluated and provided to an experimental group, but the no-intervention control group receives some special attention that can contribute to the results. - That special attention poses a threat to internal validity if it is a plausible explanation of the findings - this is more likely to occur in applied settings (schools, clinics, etc.) than in lab studies with college students.

Question 43

How is special treatment of controls a threat to internal validity?

Answer 43

The no-intervention group may be receiving an "intervention" in its own right that obscures the effect of the program provided to the experimental group. - Special treatment is different from "no treatment" and might interfere with the outcome. - Special treatment is not the same as diffusion of treatment, because essential ingredients from one group do not spread to another.

Question 44

What other influence that is part of special treatment is a threat to internal validity?

Answer 44

When participants are aware that they are the control group, they may react in ways that obscure the differences between treatment and no treatment.

Question 45

Why is it better to use "treatment as usual" compared to no treatment?

Answer 45

One does not have to worry about demoralization from no treatment and attrition caused by not getting any treatment. - Using treatment as usual at least ensures that everyone receives some viable intervention and this makes less plausible (but does not eliminate) the possibility that specialness of the intervention could explain the results.

Question 46

Under what four circumstances do threats to internal validity usually occur?

Answer 46

1. Poorly designed study 2. Well-designed study but sloppily conducted 3. Well-designed study with influences hard to control during the study 4. Well-designed study but the results obscure drawing conclusions

Question 47

What is an example of poorly designed studies where internal threats are more plausible?

Answer 47

A single group with a pre-post test (history, maturation, and testing effects are plausible).

Question 48

What is the purpose of doing a pilot study?

Answer 48

To get a feel for the experimental manipulation, how to do it, to ask clients about desirable and objectionable procedures, to evaluate outcome and preliminary results, etc. - A pilot study without a control is fine to work out details, feasibility, etc. It is intended to help develop a controlled study from which inferences about the intervention effect can be drawn.

Question 49

How can investigators prevent diffusion of treatment?

Answer 49

By monitoring the manipulation to ensure that it is delivered correctly to each group and there is no mix up, spill over, or lapses.

Question 50

What are good study designs that are conduced sloppily?

Answer 50

Studies in which researchers do not measure how faithfully or correctly the interventions were administered.

Question 51

What is a well-designed study with influences that are difficult to control during the study?

Answer 51

These are cases that might often include attrition, i.e. studies that include multiple sessions. - Attrition can be controlled a bit by the investigator over the course by keeping the study brief or providing incentives. In grant-funded research, money is available for participants who complete the final intervention session and the post-intervention assessment battery. - If subjects were in a control group, sometimes they are promised access to the intervention without cost, but only after they complete the second "post" assessment for the period in which they did not receive the intervention.

Question 52

In which cases do results obscure the drawing of conclusions?

Answer 52

When results show that the conditions were equally effective. - I.e. both groups have a pre to post change. In this case, history, maturation, testing, and potentially statistical regression could completely explain the results.

Question 53

What does random assignment prevent?

Answer 53

Selection biases.

Question 54

What do control/comparison groups prevent?

Answer 54

History, maturation, and testing.

Question 55

How can diffusion be controlled for?

Answer 55

1. Recording sessions and checking them to evaluate the procedures. 2. Sitting in on a session.

Question 56

What is external validity?

Answer 56

The extent to which the results of an investigation can be generalized beyond the conditions of the experiment to other populations, settings, and circumstances.

Question 57

What do threats to external validity constitute?

Answer 57

Questions that can be raised about the limits of the findings. - The demonstrated relation between the IV and the DV may apply to some people but not others.

Question 58

Only when can something be considered a threat to validity and not just a superficial criticism?

Answer 58

When it is a plausible factor that restricts the generality of the results.

Question 59

What are different types of external validity?

Answer 59

1. Sample characteristics 2. Narrow stimulus sampling 3. Reactivity to experimental arrangements 4. Reactivity of assessment 5. Test sensitivity 6. Multiple-treatment interference 7. Novelty effects 8. Generality across measures, setting, and time

Question 60

What are the different types or levels of concern in generalizing from one sample to another?

Answer 60

Human and nonhuman animals: The extent to which findings from nonhuman animal research can be extended to humans.

Question 61

What does animal research specifically do to get as close to human as possible?

Answer 61

They try to identify critical mechanisms that might be involved in a problem where applicability of key findings to humans are already known.

Question 62

What is the problem with using college students as subjects?

Answer 62

College students are often WEIRD. And WEIRD samples do not necessarily represent individuals from other cultures in fundamental ways in such areas as attributions, reasoning style, personality, etc.

Question 63

How do college students jeopardize psychology as a field?

Answer 63

It prevents us from generalizing many of our findings. - Studies with college students are rarely replicated with other populations. Yet, we know that there are important cultural differences in how people view the world and see their places in nature, and that such frameworks affect memory, reasoning, perception, etc.

Question 64

What are samples of convenience?

Answer 64

The selection and use of subjects merely because they are available.

Question 65

What is the most common use of samples of convenience?

Answer 65

Situations in which a sample is recruited for and well-suited to one purpose.

Question 66

What are underrepresented groups?

Answer 66

The limited inclusion of women and underrepresented and minority groups as research participants. - Historically, women and various ethnic groups were not extensively studied in the context of many topics in biological, behavioral, and social sciences in the U.S.

Question 67

What is the purpose of extending research to different groups?

Answer 67

To understand the processes, sources of influences, and factors that might dictate why or how a finding in one way in this context but another way in a different context.

Question 68

What do stimulus characteristics refer to?

Answer 68

Features of the study with which the experimental manipulation or condition may be associated and include the experimenters, setting, interviewers, or other factors related to the experimental arrangement. - Any of these features may restrict generality of the findings.

Question 69

What question does narrow stimulus sampling lead to?

Answer 69

Whether the results will generalize to more than one stimulus condition provided in the study.

Question 70

What is a concern when narrow stimulus sampling occurs?

Answer 70

Whether the effectiveness of treatment in well-controlled studies is greater than the effects obtained in clinics outside of the lab. - Once one leaves the controlled setting, the treatments tend not to be administered with the same care, which might be a reason why effects do not transfer.

Question 71

What might be done to restrict effects of narrow stimulus sampling?

Answer 71

Evaluating the impact of the treatment across two or more different types of settings in the same study. - Only done occasionally, due to logistics of implementing interventions in even one type of setting.

Question 72

What is reactivity to experimental arrangements?

Answer 72

Being aware of participating and responding differently as a result. - This leads one to question whether the results would be obtained if the subjects were not aware that they were being studied.

Question 73

How can we assure that reactivity will have less of an impact on the experiment?

Answer 73

1. Having no observers (humans) around 2. Collecting data automatically (machine) or by camera.

Question 74

How is reactivity a matter of degree?

Answer 74

Subjects can vary in the extent to which their performance is altered and experimental arrangements can vary in the extent to which they are likely to foster special reactions.

Question 75

When can reactive arrangement occur?

Answer 75

Whether or not subjects can explicitly state they are in an experiment, given that we often respond to cues that are out of our conscious awareness.

Question 76

What is reactivity of assessment?

Answer 76

Focuses on the measurements and measurement procedures.

Question 77

When are measures obtrusive?

Answer 77

When subjects are aware that their performance is being assessed. - Obtrusive measures are of concern in relation to external validity because awareness that performance is being assessed can alter performance from what it would otherwise be.

Question 78

What is the main strategy for combatting reactivity?

Answer 78

Including a measure where the purposes are not so clear and where distortion is less likely. - I.e. by using the IAT.

Question 79

Which two major forms of reactivity might affect the generality of the results?

Answer 79

1. The experimental arrangement: Whether subjects believe they are participating in an experiment or might pick up cues that guide their behavior whether or not they are aware. 2. The assessment procedures and whether the subjects are aware of being assessed and presumably can alter their performance as a result.

Question 80

What is a drawback of pre-testing participants (i.e. pretest sensitization)?

Answer 80

A pre-test might in some way sensitize the subjects so that they are affected differently by the intervention. - Individuals who are pretested might be more or less amenable or responsive to an intervention than individuals who are not exposed to a pretest merely because of the initial assessment.

Question 81

What is the difference between testing effects and pretest sensitization?

Answer 81

Testing effects: Improved score on a test is a function of completing measure more than once. Pretest sensitization: Pretest + intervention led to a special outcome at posttest. - The intervention might not show that same effect if it were not for the pretest that helped increase the impact of the intervention.

Question 82

What is multiple-treatment interference?

Answer 82

Drawing conclusions about a given manipulation or intervention when it is evaluated in the context of other manipulations. - The threat refers to instances in which subjects receive more than one condition in an experiment.

Question 83

Why is multiple-treatment interference a threat to external validity?

Answer 83

Because the conclusion about one intervention may be restricted to those circumstances in which prior manipulation or intervention was not provided. - I.e. the effects obtained in the experiment may be due in part to the context or series of conditions in which it was presented.

Question 84

How can multiple-treatment interference be prevented?

Answer 84

By having a group experience condition A and then condition B, and a group experience condition B and then condition A. By doing this, one can evaluate whether a given treatment has different effects with and without potential interference of prior treatment.

Question 85

What are novelty effects?

Answer 85

The possibility that the effects of an experimental manipulation may in part depend upon their innovativeness or novelty in the situation. - The effects of the intervention may depend upon and be limited to the context in which it is administered.

Question 86

When are novelty effects likely to emerge?

Answer 86

When some intervention/manipulation is compared to no-treatment/no manipulation. - The study seems controlled, but the intervention/manipulation may have worked because of its novelty that in fact was not controlled.

Question 87

What threats to external validity exist regarding measures?

Answer 87

1. Whether results obtained on measures may be unique to those particular measures rather than to carry over to other indices of the construct. 2. Do the intervention effects carry over to other indices of how individuals are doing in daily life>

Question 88

What threats to external validity existregarding setting?

Answer 88

1. The extent to which findings are likely to be restricted to that setting. 2. Whether the results generalize to other settings that perhaps have more impediments that could limit effects of the program.

Question 89

What are threats to external validity with regard to time?

Answer 89

1. Whether findings are restricted to a particular point in time. 2. Whether these effects would be evident for one month or one year after treatment.

Question 90

What is a cohort?

Answer 90

A particular group of people who have shared something over a particular time period. - Usually refers to a group born in a particular period or a group that is studied but gathered at a particular point in time and followed.

Question 91

What are potential problems with cohorts?

Answer 91

Whether the finding generalizes to e.g. other periods in which the relation would be assessed for these same subjects.

Question 92

What is a proof of concept?

Answer 92

A test to see whether something could occur that is important in principle or in theory.

Question 93

What is one way of conceiving many of the threats that were mentioned?

Answer 93

The notion of context: It might be that the experimental manipulation or intervention achieved its effects because of something about the context in which it was studied or demonstrated. - In this case, one can raise the question of external validity.

Question 94

How are parsimony and plausibility pertinent to the threats to validity?

Answer 94

Key threats to internal validity are threats in part because they are often parsimonious interpretations of the data and often as or more plausible than the interpretation proposed by the investigator.

Question 95

Why is internal validity usually a priority compared to external validity?

Answer 95

One must first have an unambiguous finding before one can ask about its generality. - A well-conducted study with a high degree of internal validity may show what can happen when the experiment is arranged in a particular way.

Question 96

What question does failure to generalize lead to?

Answer 96

The question of why it did not generalize. In the process, a deeper level of understanding of the phenomenon of interest is possible. - It may be that the relation depends on the presence of a third variable or an artifact in the experiment. - Thus, establishing when the finding does and does not hold can be a conceptual advance.

Question 97

When do issues of external validity emerge?

Answer 97

When there is a failure to replicate a finding.