Experimental and Non-Experimental Designs Flashcards
Hallmarks of Experimental Design
- Used to Assess causality
- Manipulation of an independent variable
- Random assignment of subjects to groups
- Control of extraneous variable
- Controls help to eliminate alternative causes
- Two main types: Between vs Within
Hallmarks of Quasi-Experimental Design
- Include some kind of intervention, manipulation, or treatment but not the same degree of control found in experiments
- Some attempt at “control”
- Lack of randomisation
- Need to address the shortcoming as closely as possible
- E.g., School-based intervention
Hallmarks of Non-Experimental Research
- Examine nature of observed relationships between groups (e.g. pubertal timing and grade difference)
- No manipulation of variables
- Use of pre-existing and intact groups
Hallmarks of Correlational Research
- Examine nature of observed relationships between two variables (e.g. body dissatisfaction and self-esteem)
- No manipulation of variables
Strengths of Non-experimental and Correlational Research
- Descriptive
- Non-Intrusive
- High external validity (variables as they exist in the real world)
Weaknesses of Non-Experimental and Correlational Research
- Cannot assess causality
- Third variable problem
- Directionality
- Low internal validity (little to no control of possible confounding variables)
Third Variable Problem
Although a correlation may be found, the relationship can be explained or is also influences by other variables.
E.g., The relationship between Body Dissatisfaction and Binge Eating can also be explained/influenced by variables such as depression, and dieting behaviours etc.
Directionality Problem
Can not determine the leading variable in a correlation.
E.g., Body Dissatisfaction and Depression, can not determine which leads to the other, or if it is bi-directional
Descriptive Research
- Describing Individuals
- Observational research
- Case studies
- Means, modes, averages usual measure
Internal validity
“the degree to which the study accurately answers the questions it was intended to answer” - Gravetter and Forzano (2012)
Extraneous Variable
Any variable that a researcher is not directly interested in
The three main types of extraneous variables are:
- Environmental (e.g., time of testing, different rooms)
- Participant (e.g., gender, age, IQ)
- Time-related (e.g., weather changes, becoming fatigued)
Confounding Variable
Any variable that influences two variables being studied, it provides an alternative explanation for the relationship (re: Third Variable Problem)
Environmental Variables
Examples:
Time of Testing
Different experimenters (such as mood studies, some experimenters may be more influential)
Assignment Bias
E.g. Use of intact groups
groups may vary in participant characteristics (gender, smokers vs non smokers)
Examining Groups over time
Changes in participants may be due to some other factor
i.e. History or maturation
Regression towards the Mean
Extreme scores on first testing tend to be less extreme on second testing
*is a statistical issue*
External validity
“refers to the extent to which we can generalise the results of a research study to people, settings, times, measures, and characteristics other than those used in that study” Gravetter and Forzano (2012, p.168).
Three main kinds of generalisations include:
- Generalisation from sample to the general population
- Generalisation from one research study to another
- Generalisation from a research study to a real world situation
Threats to Internal Validity
threat to internal validity is any aspect of the research which “raises doubts about the limits of research results or about the interpretation of the results.”
e.g. Two of the main threats to internal validity include environmental variables and assignment bias. Five other main threats to internal validity that need to be evaluated for designs that compare groups over time include history, maturation, instrumentation, testing effects, and regression toward the mean.
Threats to External Validity
Threat to external validity “is any characteristic of a study that limits the generality of the results.”
Three main kinds of generalisations include:
- Generalisation from sample to the general population
- Generalisation from one research study to another
- Generalisation from a research study to a real world situation
In addition, three general threats to external validity are:
- Participant characteristics (e.g., selection bias, restricting participants to age)
- Features of the study (e.g., reactivity, experimenter characteristics)
- Measurements (repeated measurement, use of specific measures)
Novelty Effect
Acting differently in a new situation
Reactivity
Acting differently when being observed or tested
Multiple Treatment Interference
Participant may have taken part in other treatments/programs
Repeated measurement (sensitization)
Completing a measure multiple times can lead to natural changes
e.g. practice, & fatigue
Experimenter Characteristics
Will you get the same finding with another experimenter
Treat to both internal and External and Internal Validity
Experimenter Bias & Demand Characteristics
- honesty or self-deceptions of researchers themselves
- These threats include fraud, deception and self-deception. While generally unlikely, there is the possibility of a researcher making up data and therefore outright fraud, but also the probably more common possibility that the research is unintentionally “slanted” through means such as data fabrication, hoaxes, misrepresenting data, not reporting necessary study design details, data mining (analysing data without ‘a priori’ hypotheses until a significant finding emerges), and publication bias (only publishing studies with significant findings).
Ways to Control Extraneous Variables
- One way of controlling for environmental extraneous variables is by holding these constant
- Another way of controlling all three types of extraneous variables would be to balance or match these across each level of the independent variable (For example, a researcher can ensure that there are the same number of males in each treatment. If the researcher has two testing rooms, then an equal number of participants from each treatment need to be tested in the two rooms. Similarly, in order to control for time-related variables, a similar number of participants from each treatment can be tested early and others can be tested later.)
- The final method for controlling participant variables is the use of random assignment. When subjects are randomly assigned to groups then there is a greater likelihood that participant variables will be distributed evenly across the groups and any participant differences between groups should even out.
Between-subjects designs
(Independent-measures design)
- Mets criteria of experimental design
- Participants take part in only one treatment, and one treatment only
- Can have multiple groups for each type of treatment, and will have a control group
- Advantages: Always an option, no practice or fatigue (etc) across multiple treatments, no carryover effects
- Disadvantages: require larger numbers, large individual differences can obscure treatment effect
Within-Subjects Design
(Repeated-Measures design)
- Meets experimental design criteria
- One group of participants, and all participants take part in each of the treatment groups
Main advantages of within-subjects designs:
- They require fewer participants
- They eliminate the variability due to individual differences, which in turn makes the design more powerful (more likely to detect a treatment effect)
Main disadvantages of within-subjects designs:
- Within-subjects designs cannot be used effectively if there are carryover effects (that is if doing one condition will alter the way subjects perform in another condition-e.g., long lasting drug effects)
- Within subject designs are subject to time-related factors (e.g., fatigue, practice)
- Within-subjects designs may not be practical if that study extends over a long period of time (e.g., too demanding for subjects, participant attrition)
How to control for Carry-Over Effects
In order to control for carry-over effects, an experimenter could use a between-subjects design. However, order effects that may be due to practice or fatigue can be controlled by counterbalancing. Counterbalancing involves administering treatments in a different order “so that every treatment has some participants who experience the treatment first, some for whom it is second, some third, and so on” (Gravetter & Forzano, 2012, p. 266). Because participants complete the treatments in a different order, any effect of the independent variable cannot be attributed to the timing or order of the conditions. In addition, any time-related effect would be balanced out across the conditions.
One type of counterbalancing is complete counterbalancing. In complete counterbalancing the treatments are presented in every possible order. However, this becomes impractical if you have more than four treatments.
A more practical method is partial counterbalancing. Partial counterbalancing uses different orders to ensure that each treatment occurs in each position (first, second and so on). It is also essential that each different order is used the same number of times (i.e., participants need to be in multiples of the number of treatments). Finally, to also eliminate any order effects, you need to ensure that each condition does not always follow or precede another.
Developmental Research Designs
- Study changes in behaviour in relation to age
- Cross-sectional research design
- Longitudinal research design
- Cross-sectional longitudinal design
- Longitudinal-sequential design
Cross-sectional design
- Basically a variation of the correlational design but independent groups are specifically defined by different ages. Another distinguishing feature of this design is that the different age groups are tested at the same time.
- It is like differential research
- E.G. measure moral reasoning between grade 3, 5 & 7, rather than multiple measures over time
Strengths: Time efficient, no selective attrition (participant drop-out), no practice effects
Weaknesses: Individual changes not assessed, cohort effects between different ages
Longitudinal Research
- The longitudinal developmental research design involves testing participants more than once over time. The time period can involve months or years.
- Same group of participants examined over time, and they are tested more than once
Strengths: Assesses individual behaviour change in same cohort, allows a study of antecedents and consequences
Weaknesses: Time consuming, biased sample, attrition bias or selective attrition (drop out), practice effects, only one cohort so can’t generalize to cohorts from another time or context, changes in theory and method
Cross-sectional Longitudinal
- Tests different cohorts (each tested once) at different time points to observe changes in variable over time
Longitudinal-Sequential Design
This is a special and rare type of design which includes a combination of the longitudinal and cross-sectional designs. More specifically, it includes at least two cross-sectional groups which are followed longitudinally.
The main strength of this type of design is that it permits both longitudinal and cross-sectional comparisons. However, it also allows us to detect any cohort effects as we can compare participants of the same age who were born in different years.
