Competence I: Ch 5, 7, & 8 Flashcards
Alternative explanations for study results
to draw valid conclusions about research question, the researcher must design a study that minimizes potential to generate alternative explanations for the studies results. Four major inferences made by researcher when evaluating validity of a design to avoid alternate explanations.
Statistical Conclusion Validity
the degree to which the researcher has come to the correct conclusion about the relationship.
Internal Validity
degree of certainty with which one can make statements about the existence of a causal relationship between variables.
Construct Validity
how well the variables chosen to represent a hypothetical construct actually capture the essence of the hypothetical construct.
External Validity
refers to the degree to which the causal relationship is generalizable across units, treatments, outcomes, and settings.
Philosophy of Science
conceptual specifics of validity and threats to validity are not concretely fixed as innate properties of research. These concepts are deeply embedded in the philosophy of science and evolve over time as philosophers think what it means to conduct research and make inferences about their findings.
Research Consumers
should examine threats independently because publication of results does not ensure that conclusions are valid. The DARE program is an example of this- generally believed effective when in fact not
Null Hypothesis
predicts that there is no relationship between the variables
Plausibility of a threat to validity
most threats discussed are possibly present in a study. The validity of a conclusion is suspect if a threat is plausible and the threat created conditions hat could have produced evidence supporting the conclusion.
-ex: significant difference in treatment effectiveness for volunteers versus nonvolunteers. However nonrandom assignment is a plausible threat and also could have explained the effectiveness of treatment found for the volunteer group
Low statistical power (Type II Error)
Incorrectly conclude there is no relationship when in fact one does exist. Major reason for this is that variability in the participants responses tend to obscure true relationships. This is also called error variance. Low power often comes from too few participants
Pretest
test given before manipulation. Can provide baseline or starting point to assist in measuring progress and growth. Also can be used to ensure groups are both at same starting level.
Time-Series Designs
multiple observations over time. Observations can involve same participant or similar participants. Interrupted time series design treatment is administered at some point in the series of observations. The point at which treatment takes place is interruption of series. Compare observations before and after treatment. (quasi-experimental)
Baker, Johnson, Kopala & Strout
Experimental post test only control group design in a laboratory setting (observe fake interpretation of client in a lab). Random assignment of participants to 3 conditions. Analogue b/c of the rehearsed fake client. Delayed control like waiting list control group
One-shot Pretest/Posttest Design
O1 X O2
Nonequivalent group post test only design
O2
Randomized post test only Design
R X O1
R O2
Determining tradeoffs in design and implementation of a study
designs that increase the certainty of both causal inferences (internal validity) and statistical conclusion validity may decrease the certainty of generalizing inferences from samples to populations (external validity) or meaning of the operations (construct validity). Tradeoffs occur with different types of research designs not only with regard to validity but also other factors.
Statistical tests in research studies
statistical test is used to examine whether there is a relationship between the variables in a study
Alternative Hypothesis
there is some true relationship between the variables in the study.
Statistical Significance
When a statistical test suggests chances of incorrectly concluding a relationship are fewer than 5 out of 100, 1 out of 100, or 1 out of 1000
P values and probability theory
statistical tests are based in probably theory and used to indicate whether you should reject or accept the null hypothesis. You would reject the null hypothesis is p<.05. The chances of incorrectly concluding that a true relationship exists are fewer than 5 in 100. Statistical testing of null hypothesis is embedded in probability theory and the philosophy of science and remains source of controversy.
Population
everyone fitting certain predetermined characteristics in the entire human species. This is what you are attempting to generalize your study results to.
Sample
The individuals in the study that have predetermined characteristics. You want your study to be representative of the diverse nature of the population so you can generalize your results to this population.
Type I Error
Incorrectly concluding that a true relationship exists
Unreliability of Measures
introduce error variance and obscure true state of affairs. Threat to statistical conclusion validity. Ex: bathroom scale gives dramatically different weigh each time you use it. Unlikely scores obtained from this scale will be related to other scores in any systematic way.
Unreliability of Treatment Implementation
Not having consistent implementation of the treatment, thus need to standardize treatment condition as much as possible so each person in treatment group has identical experience. The variations lead to uncontrolled variability that obscures true relationship between IV and DV. Threat to statistical conclusion validity.
True Experimental Design
Two studies that fall under this category are Between-Groups design and Within-Subjects Design. Random assignment, manipulation of the IV. Emphasis on experimental control, minimizing extraneous variables, and internal validity.
Between Groups Design
true experimental design, , random assignment of participants , looking for differences among different groups (control versus treatment groups), tightly controlled experimental conditions. Adheres to MAXMINCON principle.
Within-Subjects Design
attempt minimize error variance due to individual variation by having each participant serve as own control because all participants are exposed to treatment. True experimental design because random assignment of treatments and manipulation of the IV. Random assignment occurs as assignment to a time period in which the treatments are delivered. Types: Crossover and latin square
Quasi-Experimental Design
Involves manipulation of one or more IV, but not random assignment of participants to conditions. Allows for more flexibility and brings more limitations in terms of internal validity. Such as random assignment or intact groups to conditions.
Random Assignment of individuals to treatments
This occurs in experimental design able to take individuals and break them up into individual groups.
Random assignment of groups to treatments
This occurs in quasi-experimental designs you have in tact groups and randomly assign each intact group to a treatment or control condition.
Posttest
test given following treatment at conclusion of study to measure effectiveness. Posttest alone cannot determine growth because without pretest no measure of starting point.
Ambiguous Temporal Precedence
If threats to internal validity can be ruled out, appear that manipulation of the IV caused change in the DV and not vice versa. However, Direction is not as clear in designs in which the IV is not manipulated. Ex: empathy of counselor cause client progress or does client progress cause counselor more empathetic?
Posttest-only Control Group Design
Random assignment of participants to two groups, one of the groups receives exposure to treatment while the other group serves as control group, and thus receives no treatment. Both groups receive posttest, but neither receives a pretest. Purpose of design is to test the effect of X (IV) on observations of the DV, O1 & O2. BGD
R X O1
R O2
Pretest-posttest control group design
Random assignment of participants to tow (or more) groups with one group receiving treatment while the other group receives no treatment (serving as control group). Both groups receive a pretest and a post test. Purpose is to test effect of the IV (X) which is reflected in differences on the DV, specifically between O2 and O4 (BGD)
R O1 X O2
R O3 O4
Solomon Four Group Design
Combination of pretest-posttest control group design and posttest only control group design. Purpose of design is to examine potential effects of pretest which is one of the main weakness or unknowns of pretest-posttest control group design. (BGD) R O1 X O2 R O3 O4 R X O5 R O6
No-treatment Control Group
Does notreceive treatment. Participants in such groups seek treatment elsewhere and therefore ineffective measure as a control
Waiting-List Control Group
Participants are randomly assigned to either treatment or waiting list control group. At the end of treatment phase and posttest. Treatment is made available to participants in waiting-list condition. Disadvantage: long term follow up of the control participants is lost b/c receive treatment. Also, although participants receive treatment, treatment is withheld for a period of time
Placebo Control Group
Participants led to believe they are receiving a viable treatment even though the services rendered are nonspecific and supposedly ineffective. Ex: allows for participants to separate out specific effects of a treatment from effects due to client expectations, attention, and nonspecific aspects.
Matched Control Group
Participants are in a matched control group and are paired in some way with participants in the treatment group. Purpose is to reduce variance due to a matching factor. Posttest is administered to control participant at same time as posttest administered to pair participant in treatment group to hold constant time of treatment.
Factorial Design
used when two or more IVs are employed simultaneously to study their independent and interactive effects on the DV. Create a grid such as a 2 by 3 and put scores from each group into it (IV 1: gender female and male; IV 2: treatment type, Treatment A, Treatment B, and Control). BGD
Random Assignment = Equivalent Groups?
Random assignment does not GUARENTEE equivalent groups, only that the differences occurring between groups occurred by chance.
Nuisance Variable
: found in dependent samples design *BGD), Variable is not interested for its own sake is not examined explicitly (does not become iV) but remains an important consideration because could affect the results in unknown ways. Ex: pretest level of psyc functioning not interested to research in effectiveness of treatment but need treatment and control groups to be comparable on psyc functioning so it does not confound the results. So match participants on basis of confounding variables and randomly assign one of matched participants to treatment and control group (Stratified sampling)
Non-equivalent Control Group Designs
Comparisons are made between or among participants in non-randomly formed groups. Groups are nonequivalent b/c participants have generally been assigned to groups before the research being conducted and therefore might differ on several characteristics before the intervention. (Includes uninterpretable nonequivalent groups & interpretable nonequivalent groups & Cohort Design) (quasi)
Interpretable Non-Equivalent Group Design
participants are nonrandomly assigned to groups and then pretested on dv. One group receives treatment and other is control. Design not involve treatment control group comparison, involve comparison of two or more active treatments. (quasi)
Uninterpretable non-equivalent group design
is called “un-interpretable” because of multiple threats to internal validity,” (quasi)
Cohort Designs
case of non-equivalent group designs that utilize adjacent cohort groups that share similar environments. Allow researchers to make causal inferences because comparability can often be assumed between adjacent cohorts that do or do not receive a treatment (Ex: first grade class in 2003 & 2004). Cohorts are more likely to be similar to each other than in typical nonequivalent group designs. (quasi-experimental)
Baker, THomas, and Munson
quasi-experimental because class members cannot be randomly assigned intact groups. Do randomly assign each class to experimental or control condition. Nonequivalent control group or cohort type design. Field study b/c ninth grade classroom in public school. What makes it an interpretable nonequivalent control group design?
