Experimental Design Flashcards
Negative Control
Group added where you know the outcome but the IV will not affect the group
- Often considered the best control
- Helps guard against epiphenomenon (result that accompanies another, but has no causal influence itself or what not caused by the experiment) – an observed effect in this control means there’s something else influencing the DV.
- Known to give a - result.
Positive Control
Group added to a design where you know what the outcome will be, and the outcome is expected to move in the direction you think the IV should move it
- Used to ensure the experimental set up is working
- Provides comfort about the effects of the IV
- Similar to the actual experimental test, but which is known to give a + result
Vehicle Control
Injectate or pill without the substance (placebo) Contains everything you are administering except the level of IV
Sham Control
Generally associated with a surgical procedure, in while a mock surgery is performed. Form of procedural control.
Procedural control
Running the same procedure without the active intervention.
*One group has one procedure, other group has slightly altered procedure w/o the IV
Repeated measures
Use the subject as their own control, or alternatively using one side of the animal as a control for another animal
*This method can sometimes introduce possible learning curves/practice effect
Hypothesis driven
Research in which a specific hypothesis is laid out upfront and then tested prospectively
Prospective study
Hypothesis laid out and research is analyzed based off data obtained in the future, after application of the IV.
Retrospective studies
Assess effect of an IV after the fact.
Discovery research
formulates the basis for hypothesis driven research
*Generates Ho
Clinical Trial
study of a group of individuals that have something is common and who are assessed as they move forward in time (Prospective study)
**Usually assessed more than once (Repeated measure).
Clinical Trial
A study on patients (human) that is prospective, and tests a very SPECIFIC question- generally about a drug or specific intervention.
*Is the most rigorous (from control perspective)
Cohort study
study of a group of individuals that share something in common, and who are assessed as they move forward in time (prospective)
- Greater work load
- Over a time period
- Usually repeated measure.
Cross-sectional study
Similar to a cohort study, except that all the measures are taken at the same time (one point in time, usually present) “snap-shot” of your test population at specific time point.
- Retrospective study
- Prevalence study
Case-control study
Similar to cross-sectional study, except that these are looking at various past times– identifies one group with variable, one without said variable, and assesses their PAST habits/lifestyles in order to formulate an association.
- Retrospective
- Past focused
Ethnographic research
Study of human behavior in natural context, involving OBSERVATION of behavior in physical setting
Explanatory Research
Where the experimenter seeks to determine cause and effect. NOT and association.
*Most studies look at associations between IV and DV, not cause and effect
Historical Research
The systematic collection and evaluation of data relation to past occurrences in order to describe causes, effects, and trends of those events that may help explain present events and anticipate future events
Quasi-experiment
*Empirical study
used to estimate the causal impact of an intervention on a target population.
*Share many elements with traditional experimental design, but lack the element of random assignment to control or treatment
*The investigator does not have control over the assignment of the IV as true in other experimental design – but is not random, some control through other methods (ie, eligability cut-off)
Prevention Trials
Look for better ways to prevent disease in people who do not have the disease
*Outcome research
Screening Trials
Test the best way to detect a certain disease of condition,
Diagnostic Trial
Conducted to find better tests or procedures for diagnosing a particular disease/condition
Increases diagnostic value
Treatment Trial
tests experimental treatments, new combinations of drugs, or new approaches to surgery/radiation therapy
Quality of Life Trials
(Supportive Care trial) Explores ways to improve comfort and quality of life for individuals with chronic/terminal illness
Compassionate Use Trials
“Expanded Access Trials”
- Provide partially tested, unapproved therapeutics to a small amount of patients who have no other realistic options for treatment. Usually this involves a disease for which no effective therapy exists, or a patient who has already exhausted all other available options without success. Health must be so declined that they do not qualify for other randomized clinical trials.
- Case-by-case approval from the FDA and pharma company
Nominal data
There is no inherent value in the number, is simply associated with a group or outcome
Eg: Group 1, blood type AB
Ordinal data
Sense of a higher number reflecting something greater, but the difference b/w 1-2 may be different than that of 3-4.
If the scale is big enough the differences between the values become less different and the scale becomes more continuous.
Continuous Data
*Most informative
Intensity or measurement increases in a linear fashion, indicative by the magnitude of the difference.
P-value
Relative statement of the probability of how different two sets of data are, based on chance.
*Usually accept p-value of under .05
Observer Bias
Bias of the experimenter, based on expected outcome or intended results.
*Fix is double-blind experiment
Instrument Bias
Instrument outcome varies by factor
Subject Bias
Bias of the test system. Usually observable in clinical trials, where patients that know/think they are receiving a drug will have fabricated effects.
*Placebo effect - opioid system
4 types of Variability
- Within group
- Between group
- Within Subject
- Between Subject
Goal of the Design
Minimize: between subjects, within subjects, and within group variability
Maximize: Between group variability (Effect Size)
*Determines probability
The less variability within groups and subjects, the easier it is to determine a statistical effect.
Sample
A proportion of a population chosen to reflect characteristics of the population as a whole.
Sampling error
Selection of a sample that is bias, or not representative of a population as a whole
Convenience Sampling
Occurs when one selects a sample based on their accessibility
EX: choosing college students for a university study, picking people within the vicinity of my office
Judgement Sampling
Occurs when subjects are chosen by an individual familiar with the characteristics of the population
-Choosing subjects because of a pre-determined expectation of characteristics
Random Sample
Each subject within a population has an equal and uncontrolled chance at being included in the sample
- Clearly the least bias and least subject to sampling errors
- Done by chance - random generation, card pick method, number sample
Simple Random Sample
Type of random sampling in which all have equal chance - generally generated from a computerized list or random lottery.
Systematic Random Sample
Created by selecting one subject randomly and then choosing the remaining subjects at regularly spaced, randomized intervals, until the desired number of samples are reached.
Ex: Choosing the 28th person on a list, and then choosing every 25th person thereafter until 15 people are chosen.
Stratified Sample
Grouping of subjects into some type of logical characteristics. Ex: grouping high school students by class: Freshmen, Sophomores, Juniors, and Seniors. Then choosing 20 students from each class rank randomly = Stratified Random Sample
Cluster Sampling
Variation on the Stratified sample, in which characteristics of grouping may not be so obvious and can be arbitrary.
Ex: Randomly dividing high school into 4 groups, not based on class/age or any other variable, and then randomly choosing 20 students from those clusters.
Choosing 10 Random school districts in Illinois and then surveying every freshman in those 10 districts.
Purposeful Sampling
Occurs when subjects/cases are chosen because they exhibit particularly rich characteristics that will help in identifying results.
EX: Testing new Schizophrenia drug, and sampling only patients who exhibit ALL the DSM IV characteristics of Schizophrenia and excluding those who only show some characteristics
Falsifiability
Concept used to distinguish science from nonscience/pseudoscience. A result can be disproved, means it has the possibility of being scientific.
Merton’s norms of true science
Originality, Detachment, Universality, Skepticism, and Public accessibility
Accuracy
The degree of conformity of a measured or calculated quantity to its actual (true) value.
Precision
Also called reproducibility or repeatability, the degree to which further measurements or calculations show the same or similar results.
Standard Deviation
Characterization of precision
- 3% confidence interval of the measurements
* This means that 68.3% of the data collected will fall within one SD of the mean of the normally distributed data set.
Standard Error or Standard error of the mean (SEM)
is the estimated standard deviation of the error in method. It estimates the standard deviation between the measured values and the true value
*Always smaller than the SD
Variability of the means taken from several identical experiments
Repeatability
The variation arising when all efforts are made to keep conditions constant by using the same instrument and operator and repeating during a short time period.
Reproducibility
The variation arising from the same measurement process among different instruments and operators over a longer time period.
AKA: Robustness
Referring to a measuring device of machine = “robustness”
Refers to individuals who are scoring the same observations
Inter-rater reliability correlation
Determines the reproducibility of observers
Validity
Refers to the concept that a model is doing what you think it is modeling or measuring
*Refers to the concept, notion, design, or hypothesis
Internal Validity
The degree to which the intervention being evaluated really caused the effects estimated in the study
*Cause must precede the effect or change in IV must precede change in DV – if not, the change in DV is due to something else than the IV.
Temporal Precedence
The cause must precede the effect in time. –> IV changes before the DV
Covariation
The cause and effect are related in some way
* Change in IV = a proportional change in DV
EX: Change in the dose of Crestor = lower cholesterol
Nonspuriousness
There is no plausible or known alternative explanations for the observed covariation. Spurious means false, not authentic or genuine
Most difficult to rule out!
Threats to Internal Validity
Confounding Repeated Measures Biasing Subject Selection Bias Age/Maturation Effect Regression Towards the Mean Floor/Ceiling Effects Diffusion Effect Differential Drop-outs/Catastrophic Event
Confounding
A control flaw, where a variable other than the IV participates in a change in the DV. Can be a known variable, or one that is unknown.
*Considered spurious
Repeated Measures Biasing
Where prior exposure to the IV affects the outcome of the DV.
“Practice effect”, “carry over effect”, or “history effect”
Practice Effect (Carry over OR History Effect)
The experimental subject “learns” from the first assessment and this influences the outcome of the DV.
Subject Selection Bias
Unknown attributes of the subjects contribute to the outcome of the effects observed. Using a single characteristic to assign subjects to groups allows for ignorance of other characteristics that may effect the study outcome.
*Can be avoided by the use of pre-tests to reduce within groups variability. Also, non-biased, totally random assignments to groups overcome this as well.
Age/Maturation Effect
Effects that occur in long term studies, or those that include subjects being studied during a developmentally critical period.
Regression Towards the Mean
Outliers tend to regress towards the mean during subsequent assessments. This would assume that the extremes in the DV measurements may reflect some spurious effect during the assessment. A rat is feeling sick, or a human is having a “good day”. This can be accounted for by multiple pre-testing paradigms, however this can attribute to practice effects.
Floor/Ceiling Effects
When the DV can not decrease or increase any further– the degree of covariance erodes as the floor and ceiling are approached.
Diffusion Effect
Effects of the IV on the DV spread across groups. EX: Actions/behavior of a treated subject group may influence the behavior of the control group.
Differential Dropouts/Catastrophic Events
disrupt casual inference. *Common in long-term studies.
People being exposed to a new drug in clinical trial are dropping out much more frequently than those in the control group, due to side effects. This may tell you something about the IV!
External Validity
Refers to the ability of your experiment to be generalized outside of the experimental setting. Does my experiment apply to the real world?
PEOPLE - PLACES - TIME
Parts of External Validity
Face Validity Content Validity Construct Validity Predictive Validity Concurrent Validity Convergent Validity Discriminant Validity
Face Validity
The model looks like the system under study.
EX: Streptozotocin induced diabetic mice should have the same characteristics of a person with diabetes
Content Validity
The extent to which a measure represents all facets of a given target system or disease.
Should depict an overwhelming number of the characteristics of the target system/disease
Construct Validity
Model is representing what it is supposed to represent.
Ex: Administering an exam with political questions on it in an experimental design class (low construct validity)
Predictive Validity
Ability of a model to predict characteristics of the target under normal circumstances. ie: Treating an STZ model with insulin should prevent the symptoms of diabetes
Concurrent Validity
The ability to distinguish among the targets it should distinguish among if it were valid.
*Variation of content validity
A model of metastatic breast cancer should have cancer spreading to the lung and bone, but not the liver or development of leukemia
Convergent Validity
Idea that the model has characteristics similar to other models of the same target.
*Ie: There are many models of dementia, and your manipulation should work in all models.
Discriminant Validity
Related to convergent validity, but opposite, and suggests that a model should be different from models of other diseases.
Threats to External Validity
*Subjects, setting, and time* Population Validity Ecological Validity Temporal Validity Treatment-Attribute interactions Treatment-Setting interactions Multiple Treatment interactions Pre- and Post-test sensitizations
Population Validity
Does the model being studied apply to real-world situations
ie: does studying cancer in mice really reflect what’s going on in humans?
Ecological validity
Relates to setting, does one model setting apply to other model settings as well
ie: Does assessing learning methods in small suburban schools reflect the learning methods in inner city or rural schools?