Chapter 2: Internal and External Validity Flashcards
What does “plausible rival hypothesis” refer to?
Those competing interpretations that might be posed to explain the findings of a particular study.
What are the four types of validity?
- Internal
- External
- Construct
- Data-evaluation
What is methodology?
A way of thinking that relates directly to how one thinks about a study.
What is a crucial part of designing an experiment?
Identifying its purpose and specific questions at the outset.
- This helps prioritize types of validity.
What is internal validity?
The extent to which the intervention rather than extraneous influences can be considered to account for the results, changes, or differences among conditions.
- The results can be attributed with little or no ambiguity to the effects of the IV.
What are threats to internal validity?
Factors/influences other than the IV that could explain the results.
- These are the problems that could emerge in the design or execution of the study.
What are the major threats to internal validity?
- History
- Maturation
- Testing
- Instrumentation
- Statistical regression
- Selection biases
- Attrition
- Diffusion of treatment
What is history?
Any event, other than the IV, occurring in the experiment or outside the experiment that may account for the results.
- These are events common to all subjects in everyday life.
- The influence of such historical events might alter performance and can be mistaken for an effect resulting from the experimental manipulation or intervention.
What does history not refer to?
Events that happen to all of us all of the time. Our individuals histories and experiences.
- Rather, history refers to any event that happens to virtually all of the subjects that might explain the findings of an experiment (e.g. COVID-19).
Can history occur during the experiment?
Yes, it includes events that take place during the experiment as well.
- When subjects are run in a group, unplanned events (e.g. fire drill, medical emergency of one participant) may disrupt administration of the intervention and reduce/enhance the influence performance of the participants.
What is needed for history to be considered a threat for a given experiment?
- A historical event occurred when the experiment was run.
- This even is a plausible interpretation of the findings.
What is maturation?
Processes within the participants that change over time and includes growing older, stronger, wiser, and more tired or bored.
- Can arise when there is more than one assessment occasion.
When is maturation a problem?
If the design cannot separate the effects of maturational changes from the intervention.
- That is, are changes within the individual over time a plausible explanation of the findings?
- The time frame of the study is relevant to invoking this threat. In general, maturation has greater impact over time.
What is testing?
The effects that taking a test one time may have on subsequent performance on the test.
Does repeated testing always lead to improvement?
No, and if it does, there are obvious limits so that endlessly retaking some measure will not lead to continued improvement.
How can you account for testing effects?
By using a control condition with repeated testing but without the special experimental manipulation or intervention.
Which threats to internal validity often go together?
History, maturation, and testing.
What is the threat of instrumentation?
Changes in the measuring instrument or measurement procedures over time.
What is the most common situation where instrumentation is a threat to internal validity?
When ratings are made by judges or oneself and somehow the standards or criteria for making those ratings change over time.
- Changes in the DV over the course of a study may result from changes in scoring criteria, rather than changes in what is being rated.
How can instrumentation affect substantive conclusions about changes over time in clinically and socially relevant domains?
Think about the case of ASD: at one point in history, being diagnosed with ASD was a rare occurrence. However, now diagnoses of ASD are more common.
- Instrumentation is involved: The ways in which cases are being identified (referred to as ascertainment) are much more thorough and comprehensive than it has been before. In addition, the definition of what counts as the disorder has changed.
How can the conditions or context of administration influence the nature of a measure, even when holding all of the items in the measure constant?
E.g. a study in China: the first time the participants filled in a survey, the measure was not part of the study. However, the second time they filled in the survey, it was part of a study and they had to sign an informed consent. Thus, the context changed.
How can the instrument change and be the same at the same time?
It is possible that the items remain the same, but the items may have a different meaning because of the social context of a given point in time.
What is response shift?
Changes in a person’s internal standards of measurement.
- Includes the case of judges who rate athletic performance or prisoners up for parole.
- Phenomenon can be more general any time there might be a change in values, perspective, or criteria that lead to evaluation of the same or similar situations, behaviors, and states, in a different way.
What is the hello-goodbye effect?
Before treatment, psychotherapy clients might see their lives as especially bleak or perhaps even distorted a little. At the end of treatment, they respond now by having altered their threshold for noting symptoms or seeing their lives differently even though the symptoms may not have changed.
What does response shift reflect?
A change in the threshold for answering a particular way.
- The threshold may be influenced by historical and maturational changes in the individual or context in which the instrument is embedded.
When is instrumentation a threat to internal validity?
In any instance in which differences might be attributed to a change in the instrument or to a change in the criteria used to complete that instrument.
What is the difference between testing and instrumentation?
Testing refers to changes in the individual over time (due to experience and practice on the measure), while instrumentation is about changes in the instrument device or how the measure is used.
- Note that both can be prevented by having a control group.
What is statistical regression?
The tendency for extreme scores on any measure to revert (or regress) toward the mean of a distribution when the measurement device is re-administered.
What are three ways to help protect against statistical regression?
- Assign participants randomly to an experimental or control condition.
What are three ways to help protect against statistical regression?
- Assign participants randomly to an experimental or control condition.
- That way, regression, if present, will affect all groups, and one can see if the experimental manipulation led to changes beyond what was evident in the control group.
- Use measures that are known to have high reliability and validity.
- Regression is a function of error of the measure. The greater the error, the more likely there will be regression.
- Test everyone twice for the pretest and select only those individuals who were extreme on both occasions.
- These people are likely to have the characteristic of interest.
- Often not feasible, which is why it rarely occurs.
What are selection biases?
Systematic differences between groups before any experimental manipulation or intervention is presented.
How are selection biases controlled for?
By randomly assigning subjects to conditions so that so that any subject characteristics that may introduce a selection bias are dispersed among groups roughly equally or at least unsystematically.
Where does selection bias often arise?
In clinical, counseling, and educational research where intact groups are selected (e.g. patients from separate clinics or hospital).
- In prevention programs, comparisons are often made among classes, schools, or school districts that do or do not receive an intervention.
When is selection bias not a threat to internal validity?
When you begin with a select group because the purpose of the study is to identify different groups and elaborate their unique characteristics.
What is attrition?
Loss of subjects, which may serve as a threat to internal validity.
When does attrition occur?
When an investigation spans more than one session.
Why is attrition a problem?
The groups of participants are no longer randomly comprised, and subjects, on the basis of variables we do not really know, elected to pull themselves out of the group.
- This could cause us to have difficulty in detecting selection bias, given that the groups are no longer randomly comprised.
How can attrition affect overall group performance?
Group performance on the measures may be due to loss of subjects who scored in a particular direction, rather than to the impact of an experimental manipulation or intervention.
Where does diffusion/imitation of treatment occur?
In any experiment where groups are exposed to different procedures.
- More likely to be a problem in intervention research. In this case, it is possible that the intervention given to one group may be provided accidentally to all or some subjects in a control group as well. This dilutes the effects of the treatment and alters what the investigator concludes about the efficacy of treatment.
