7: Specialized designs Flashcards
Quasi-experimental designs and ex post facto designs
When something is missing…: typically control groups, and/or random assignments of subjects
what are the major categories of designs
pre-experimental designs
experimental designs
quasi experimental designs
ex post facto designs
what are pre-experimental designs
no control group(s), and no random assignment of subjects to conditions. Acceptable for pilot research, otherwise avoid at all cost
what are experimental designs
control group(s), random assignment of subjects to conditions
what are Quasi-experimental designs
often control group(s), but no random assignment of subjects to conditions, i.e., use of quasi-independent variables
Do not fill the requirements for a true experimental (analytic) experiment, i.e., issues with:
* Internal validity
* External validity
Most likely issue:
* Random selection not possible or not respected (common, so often overlooked as a criterion)
* Random assignment not possible or not respected
Often take advantage of naturally occurring events. Lack of control over variables.
what are Ex post facto designs
special case of between subject design. Here we extract an association (correlation) between variables, not causation
Often take advantage of naturally occurring events. Lack of control over variables.
when are some within subject designs useful
Useful when statistical power is an issue.
Useful when few measurements are possible.
Useful when availability of participants is limited.
Typically: Random assignment of participants is not possible.
A common mistake: The “pooling fallacy”.
origin of the name ex post facto designs
In ex post facto designs, the researcher arrives “after the fact”. “Nature” has implemented the treatment (i.e., groups are based on natural occurrences that make them distinct, different):
* Different environment(s) (environmental or contextual factors)
* Different disposition(s) (dispositional or individual factors)
* Combination of these two factors
Characteristics of ex post facto designs
Participants are selected after the fact.
Two main reasons:
* Ethics (in the case of invasive studies)
* Subject variables (gender, age, educational level, personality traits, psychopathologies, etc.) become independent variables,
i.e., are treated as treatment conditions.
Rationale: You can not randomly assign people to different age groups, genders, etc, as those labels already define them as subjects.
Prospective EPF designs: internal validity issues
No random sampling from population.
No random assignment to conditions.
Confounded variables typical to the groups investigated.
* Example: High stress job participants smoke and drink more.
Selection of subjects becomes complicated and strict criteria must be applied.
Convenience sampling is problematic. The sampling criteria may themselves be confounds.
Detection bias: Common in biomedical/health research.
Accurate identification of members of the respective groups, based on criteria
Prospective EPF designs: external validity issues
Choice of groups: Can we generalize from one group (occupation) to the other?
Longitudinal nature of the study: The issue of “experimental mortality” or attrition.
what is matching in EPF designs
Making sure that the two (or more) groups do not differ on any other variable than the ones selected (e.g., high
versus low stress)
types of matching
Subject-to-subject matching
Distribution-for-distribution matching
what is subject to subject matching
Persons by person matching of subject characteristics
what is distribution-for-distribution matching
Matching on descriptive statistics, mainly central tendency or variability.
measuring the variables in EPF designs
The idea is simple: You try to identify the possible confounding variables and you measure them.
Then, we simply can determine if the variables are confounded
or not and predict their effect.
Multivariate methods and analysis of covariance (more later) can help in those situations.
Relative risk ratio as a dependent variable: avoid.
retrospective EPF designs and its comparative advantages
Very common in health research, epidemiology, etc.
Similar problems as in retrospective EPF designs + searching “backward” for information/data.
Comparative advantages:
* Smaller number of subjects required.
* Shorter time period (much shorter) required.
* Much less money required as well.
See prospective EPF’s for internal and external validity issues.
Some specific issues with retrospective EPF designs
Present context “motivating” the investigation.
Detection bias (as for the prospective EPF’s) but:
* with the addition of “historical” issues in diagnosis (related to progress in the ability to diagnose some diseases), including awareness.
* diagnosis was likely made by different parties (same “criterion” or “criteria”?).
* differential “assessment” or perspective on stress between “cancer group” and “healthy group”.
* less extensive records in the “healthy” group.
solutions for retrospective EPF designs
Same options as for the prospective EPF designs:
* Matching
* Measuring possible confounding variables
In all cases, matching or identification of measurable variables is more difficult in this case as the design requires you to search back in time.
You rely on the memory of the subjects, their family and friends, and their physicians, including the accuracy and completeness of the medical records.
Relative odds ratio as a dependent variable: avoid.
types of change
Studying changes in time of structures and processes
Ontogenetic change: Developmental research
* Changes in the individual neuron, brain, animal, person
Historical change: Cross-generational research
* Changes during across generations (strains, families, etc.)
Phylogenetic change: Evolutionary research
* Changes in species and populations: speciation, natural selection
developmental psychology: longitudinal studies disadvantages
Disadvantages:
*Test conditions become well known by the child (constantly re-tested… increased familiarity)
*Are performance and/or behavioural changes due to experience or maturation?
*Can take a lot of time (months, years…) and money, depending on the behaviours under investigation, etc.
developmental psychology: cross-sectional studies disadvantages and the solution
Disadvantage: Individual differences can account for some effects (not age per se) and (normal) differences in development
Solution: The hybrid version or “cohort sequential design” or “cross sequential design”.
comparative/ evolutionary studies
Common in:
* comparative and evolutionary neuroscience
* comparative and evolutionary psychology *
* ethology, behavioural ecology, etc.
* cross-cultural psychology, anthropology, ethnology
* linguistics
Guiding principle: Compare and contrast
Theoretical foundations: Natural selection and/or culture
- Comparative is synchronic (static in time); evolutionary is diachronic (dynamic in time, historical)
Combining within and between manipulations
mixed designs or split plot designs
* the most common cases of mixed within and between subject manipulations
nested designs or hierarchical designs
* More economical than mixed designs (within + between)
* Less information (some interactions cannot be evaluated)
objectives of the nested design
The nested design allows us to test two things:
* Difference between Control and Study (“quasi-experimental”) areas.
* The variability of the sites within areas (sites 1, 2, 3 in the Control area; sites 4, 5, 6 in the Study area).
If we fail to find a significant variability among the sites within areas, then a significant difference between areas would suggest that there is an environmental impact.
In other words, the variability is due to differences between areas and not to variability among the sites.
* BUT, in this kind of research (environmental or species monitoring), it is likely that you will find variability within the sites (within an area).
* BUT, even if you find significant difference between the sites, you can still test to see whether the difference between the areas is significantly larger than the variability among the sites with areas.
what is nested?
It depends what you are trying to accomplish, you can be
* Nesting tasks
* Nesting groups (“naturally occurring groups”)
* Nesting locations, sites, areas (“naturally occurring …”)
* Nesting times (centuries, decades, years, seasons, months, weeks, days, hours, etc.)
types of variations
A variation is to give a treatment early, but then cancel it and see the effects
Another variation is to follow the typical ABA design (or variations on this theme) that will be described in the next chapter: no treatment (A) - treatment (B) - no treatment (A)
pre-test and post-test designs
True experimental design, similar to a within-subject design.
* Two levels: pre and post treatment
Problem and solution:
* Carryovers… and no possibility to counterbalance…!
* How to increase internal validity then?
* Control groups (necessary)
* Random assignment of subjects to conditions (if, possible)
Mixed design (2X2): Pre-test / post-test is the within factor, treatment / no treatment is the between factor
solutions to the problem of the effect of having experienced the pre-test in pre-test/ post-test design
1) Eliminating the pre-test completely… simple two-group experiment
- Better:
2) Solomon four-group design: partial removal of pre-tests. This
helps you evaluate the effect (if any) of pre-tests
