Lecture 7 Flashcards
____ is the key to controlling the outcomes from experimental research.
design
6 general issues in experimental design:
- internal validity
- external validity
- frame of reference
- longitudinality
- frequency
- nested factors
Internal validity is …
the basic minimum without which any study is not interpretable
Internal validity is the extent to which the results of a study can:
- be attributed to the design of the study
- eliminate any rival hypothesis or events influencing the study
To gain internal validity, the researcher attempts to control and/or eliminate possible _____ variables.
extraneous
Internal validity is particularly important in ______ studies as well as other studies that establish _____-_____ or _____ _____.
- experimental
- causal-effect
- causal relationships
When we think the independent variable produces a change in the dependent variable, what do we have to make sure of?
- must rule out the possibility of other factors producing the observed effect
- must consider the appropriateness of the methods (participants, experimental design, measurements)
Main threat in internal validity:
unmeasured process accounting for the observed results
Secondary threat in internal validity:
responses do not correctly reflect underlying dimensions
How to minimize threats to internal validity:
- random assignment of participants into treatment and control groups
- ID confounding factors
- multiple methods approach provides evidence leading to similar results
3 main categories to threats to internal validity:
- single group threats
- multiple group threats
- social threats
Single-group threats:
a threat to internal validity that occurs in a study that uses only a single program or treatment group and no comparison or control
Multiple-group threats:
an internal validity threat that occurs in studies that use multiple groups
Social threats:
an internal validity threat that arise because social research is conducted in real-world human contexts where people will react to not only what affects them, but also to what is happening to others around them
Types of single-group threats:
- history
- maturation
- testing
- instrumentation
- mortality
- regression
History (single-group threat):
- events occurring during he experiment that are not part of the treatment
- eg. children watching Sesame Street and math score
Maturation (single-group threat):
- process within the participants that operate as a result of time passing
- eg. children mature and gain muscle strength
Testing (single-group threat):
the effects of one test on subsequent administrations of the same test
Instrumentation (single-group threat):
changes in instrument calibration, including lack of agreement within and between observers
Mortality:
loss of participants from comparison group for nonrandom reasons
Regression:
the fact that groups selected based on extreme scores are not as extreme on subsequent testing
Multiple group threats to validity is ____ ____:
- selection bias
- any factor other than the program that leads to posttest differences between groups
Multiple-group threats types:
- selection-history
- selection-maturation
- selection-testing
- selection-instrumentation
- selection-mortality
- selection-regression
Selection-history:
A threat to internal validity that results from any other event that occurs between pretest and posttest that the groups experience differently
Selection-maturation:
A threat to internal validity that arises from any differential rates of normal growth between pretest and posttest for the groups.
Selection-testing:
Selection-testing: A threat to internal validity that occurs when a differential effect of taking the pretest exists between groups on the posttest
Selection-instrumentation:
A threat to internal validity that results from differential changes in the test used for each group from pretest to posttest
Selection-mortality:
A threat to internal validity that arises when there is a
differential nonrandom dropout between pretest and posttest
Selection-regression:
A threat to internal validity that occurs when there are
different rates of regression to the mean in the two groups
Social interaction threats occur because…
- the various groups participating in the
study and the researchers carrying out the study are aware of each other’s existence and role in the study - many of these people come in contact with each other
- try to isolate
Why is it hard to randomly assign and isolate to get rid of social interaction threats?
it introduces external validity (generalizability) concerns
4 types of social interaction threats:
- diffusion or imitation of treatment
- compensatory rivalry
- resentful demoralization
- compensatory equalization of treatment
Diffusion or imitation of treatment:
Occurs because a
comparison group learns about the program either directly or indirectly from program group participants.
Compensatory rivalry:
Occurs when one group knows the program another group is getting and, because of that, develops a competitive attitude with the other group
Resentful demoralization:
Occurs when the comparison group knows what the program group is getting and becomes
discouraged or angry and gives up
Compensatory equalization of treatment:
Occurs when the
control group is given a program or treatment designed to make up for or “compensate” for the treatment the program group gets
Many threats to internal validity are controlled by making the participants in the experimental and control groups as _____ as possible.
alike
4 methods for controlling internal validity threats:
- randomization (random assignment
- placebo
- blind setup
- double-blind set up
Randomization (random assignment):
- Controls for history up to the point of the experiment but not during the experiment
- Controls for maturation because passage of time is equivalent in all groups
Placebo:
a control group receives a false treatment while the
experimental group receives the real treatment
Blind setup:
the participant does not know whether he/she is receiving the experimental or control treatment
Double-blind setup:
neither the participant nor the experimenter knows which treatment the participant is receiving
External validity threats arise when…
experimenters draw
incorrect inferences from the sample data to other individuals, settings, and past/future situations
Why do external validity threats arise?
- characteristics of individuals selected
- uniqueness of the setting
- timing of the experiment
3 major threats to external validity:
- people
- places
- times
How to increase external validity:
- Drawing on a representative probability sample and avoiding obtrusive measures
- Develop a model for the population from which one has a sample selected
- Replicate the sampling procedure and see how it compares to original sample taken
Experimental research involves comparisons of ____ performance with some _____ of performance.
- observed
- expectations
2 types of comparisons:
- relative
- absolute
Relative comparisons:
what would have occurred without the intervention or the difference between 2 of more interventions?
Absolute comparisons:
what is the degree to which the interventions reaches some particular desired outcomes?
_____ comparisons are harder to implement.
relative
____ comparisons require more advanced planning because..
- absolute
- a consensus or other evidence has to be used to compare
Longitudinality:
the extent to which measures are repeated and extended across time
Challenge with longitudinality:
mortality (ie. dropping out) over the course of time
One way to overcome mortality is…
overlapping panel designs with different cohorts
Frequency:
the number of repetitions within a time period
Higher frequency leads to…
higher costs
Nested factors:
effects which are restricted to a single level of a factor are said to be nested within a factor
In nested factors, selection of _____, _____, or _____ are not selected ______ from each other.
- locations
- individuals
- organizations
- independently
In nested factors, there may be an interaction effect between …
treatment within the embedded groups
____ ____ ___ may be the most powerful means of generating new knowledge as their confidence in the ____ ____ _____ relationship.
- true experimental design
- cause and efffect
True experimental design is typically conducted in a _____ setting.
- laboratory
- controlled environment
True experimental design has at least one _____ group (also termed _____ or ____ group).
- experimental
- treatment
- intervention
True experimental design also has at least one _____ group.
control
In true experimental design, membership is typically defined using _____ _____.
random assignment
Any study that is designed with …. is considered a true experiment.
- random assignment of participants
- a control group
- manipulation of the independent variable
A true experimental study design allows researchers to ____ data and test their ______ with a minimum _____ or ______ from _____ variables.
- collect
- hypothesis
- influence
- contamination
- extraneous
In true experimental study, ____ validity is high but ____ validity is hard to claim. Why?
- internal
- external
- tightly controlled settings will have a harder time generalizing to a larger setting
R =
random assignmnet
O =
observation of DV
X =
treatment by IV
A1, B1, A2, B2 =
notation for multiple IV, moderator, or control variables in factorial design
Quasi-experimental studies are those that are …
“sort of “ experimental in design
The big distinguishing feature between a true experiment and a quasi-experimental design is that…
there is no randomization to groups in a quasi-experimental research design
Quasi-experimental studies are low on ____ validity because…
- internal
- groups are not likely to be equivalent at the outset of the study
Quasi-experimental studies are often higher on _____ validity than true experiments because…
- external
- people are studied in a natural environment, making generalizability more feasible
The basic formula for a quasi-experimental study is:
- people are studied in real-word settings
- an independent variable is introduced or manipulated
- there is a dependent variable (the effect) that is measured.
Xa, Xb, Xc =
levels of the IV
C =
groups assigned according to a cutoff score
Repeated measures design is a specific type of ____ ______ design where there is …
- quasi-experimental
- only an experimental group with multiple levels of the independent variable
In repeated measures design, _____ are often ____ _____ control group.
- participants
- their own
Strength of repeated measures design is that it is …
economical in both time and effort of the researcher since the volunteer participants are involved several times with the study
In pre-experimental design, researchers examine …. and provide an _____ during the study.
- one group of individuals
- intervention
In pre-experimental design, there is either an _____ of a control group for ______ or a ____ _____ _____ group is identified during the study or after the intervention.
- absence
- comparison
- non-equivalent comparison
Pre-experimental research design pose challenges to both _____ and _____ validity due to …
- internal
- external
- not being able to control for many of the threats
Non-experimental research design has:
- no randomization to groups
- no groups to randomize to
- no manipulation of the IV
Non-experimental research design relies on researchers’ ______ and ______ to arrive at conclusions.
- interpretation
- observations
Non-experimental research design is used to report ______ and ______ among variables of interest.
- correlations
- associations
Non-experimental research design generally looks at _____ research problems such as…
- descriptive
- cross section (participants assessed at one point in time)
- longitudinal (following participants over time)
