Research Methods and Study Design Flashcards
Experimental Design
the technical term for a specific type of research
Steps to good experimental design
1) select the population
2) operationalize the independent and dependent variables
3) carefully select the control and experimental groups
4) randomly sample from the population
5) randomly assign individuals to groups
6) measure the results
7) test the hypothesis
1) Selecting the population
- Objective: determine the population of interest and consider what group will be pragmatic to sample
- Common Flaws: the population is too restrictive, sampling all individuals of interest is not practical
2) Operationalize variables
- Objective: determine the independent and dependent variables, specify exactly what is meant by each, make sure the dependent variable can be measured quantitatively within the parameters of the study
- Common Flaws: insufficient rigor in the description, manipulation of the independent variable presents practical problems
Dependent Variable
variable that is measured
Independent Variable
variable manipulated by the research team
Operational definition
Specification of precisely what they mean by each variable
Reproducibility
Quality of good experimental design, experiments can be reproduced by others. researchers
Quantitative
numerical
Qualitative
descriptive
3) Divide into groups
- Objective: carefully select experimental and control groups, homogenize the two groups, isolate the treatment by controlling for potential extraneous variables
- Common Flaws: control group does not resemble treatment along important variables, the experiment is not double-blind, participants can guess the experiment allowing a placebo effect to occur
Experimental Group
group of participants that receives treatment
Control group
group of participants that acts as a point of reference and comparison
Homogenous
a control group that is the same throughout and as similar as possible to the experimental group except for the treatment
Extraneous (or confounding)
variables other than the treatment that could potentially explain the results of an experiment
Placebo effect
believing that the treatment is being administered can lead to measurable results
Double blind
neither the person administering. treatment nor the. participants truly know if they are assigned to the treatment or control group
4) Random sampling
- Objective: make sure all members of the population are represented, ideally each member has an equal chance of being selected, meeting these criteria is often not possible for practical reasons
- Common Flaws: sampling is not truly random, sample does not represent the population of interest
Sampling bias
if it is not equally likely for all members of a population to be sampled
Selection bias
more general category of systemic flaws in a design that can compromise results, another example is purposefully selecting which studies to evaluate in a meta-analysis
Meta-analysis
big-picture analysis of many studies to look for trends in the data
Attrition
another type of selection bias, occurs when participants drop out of the study. If participants dropping out is non-random, this might introduce an extraneous variable
5) Random assignment
- Objective: individuals who have been sampled are equally likely to be assigned to treatment or. control, consider matching along potential extraneous variables which have been pre-selected
- Common Flaws: groups are not properly matched, assignment is not perfectly random
Randomized block technique
researchers evaluate. where participants fall along the variables they wish to equalize across experimental and control groups. Then randomly assign individuals from these groups so. that the treatment and control groups are similar along the variables of interest
6) Measurement
- Objective: make sure measurements are standardized, make sure instruments are reliable
- Common Flaws: tools are not precise enough to pick. up a result, instruments used for measurements are not reliable
Reliability
means that they produce stable and consistent results, measure what they’re supposed to (construct validity) and that repeated measurements lead to similar results (replicability)
Psychometrics
study of how to measure psychological variables through testing
Response bias
another concern with surveys, defined as the tendency for respondents to not have perfect insight into their state and provide inaccurate responses
Between-subjects design
the comparisons are made between subjects from one group to another
Within-subjects design
compare the same group at different time points
Mixed methods research
any combination of different research techniques, such as within-subjects and between subjects, or qualitative and quantitative
7) Test the hypothesis
- Objective: use statistics to check for a significant difference, assign a pre-established threshold at which the null hypothesis will be rejected
- Common Flaws: small sample size leads to insufficient power, researchers do not set thresholds in advance and make after-the-fact conclusions that lead to logical fallacies
Type 2 error
incorrectly conclude that there is no effect (false negative)
Type 1 error
falsely suppose the veracity of a result that does not actually exist (false positive)
Null hypothesis
assume that there is no causal relationship between the variables and any effect that they measure, if there is one, is due to chance
Experimental hypothesis
the proposition that variations in the independent variable cause changes in the dependent variable
P-value
number from 0-1 that represents the probability that a difference observed in an experiment is due to chance
- lower p-values suggest a stronger relationship
Sample size
number of participants
Power
ability to pick up an effect if one is actually present
External validity
flaw or limitation that might make it difficult to apply conclusions to the real world
Internal validity
Extent to which. the outcome variable is due to the intervention. A limitation that the experiment is not “well done”, leaving doubts about the conclusions because of some inherent flaw of the design.
- internal validity is high if confounding variables have been considered and minimized, and the causal relationship between independent and dependent variables can be established by the way the experiment was set up
Predictive validity
does the test tell us about the variable of interest
common threats to internal validity
impression management confounding variables lack of reliability sampling bias attrition effects demand characteristics
Impression management
participants adapt their responses based on social norms of perceived researcher expectations; self fulfilling prophecy; methodology is not double-blind
Hawthorne Effect
Confounding Variables
extraneous variables not accounted for in the study; another variable offers an alternative explanation for results; lack of a useful control
Lack of Reliability
measurement tools do not measure what. they purport to, lack consistency
Sampling Bias
selection cirteria is not random, population used. for sample does not meet conditions for statistical test (e.g. population is not normally distributed)
Attrition Effects
participants fatigue; participants drop out of study
Demand Characteristics
participants interpret what the experiment is about and subconsciously respond in ways that are consistent with the hypothesis, respond in ways that match how they are expected to behave
The experiment doesn’t reflect the real world
laboratory setups that don’t translate to the real world, lack of generalizability
Selection criteria
too restrictive of inclusion/exclusion criteria for participants (i.e. sample is not representative)
Situational effects
presence of laboratory conditions changes outcome (e.g. pretest. and post-test, presence of experimenter, claustrophobia in an MRI machine)
Lack of statistical power
sample groups have high variability; sample size is too small
Ethical consideration
to be sure ethical standards are met, modern experiments must be cleared by an independent internal commission and contain some type of disclosure
Disclosure
an outline given to participants before the experiment begins that clarifies incentives and expectations while reminding them of their right to terminate the experiment at any time
Debriefing
participants are told after the experiment exactly what was done and why the experiment was conducted
Non-Experimental Designs
Correlational studies ethnographic studies twin studies longitudinal studies case studies phenomenological studies survey archival studies biographical studies observational studies
1) Correlational studies
- Description: measures the quantitative relationship between two variables
- Strengths: great. preliminary technique, usually easy to conduct
- Weaknesses: does not establish causality, may not pick up non-linear relationships
Pearson correlation
assigns a number from -1 to +1 to a pair of variables
- if the value is negative, the two variables are negatively correlated (if one increases the other decreases and vice versa)
- positive value represents a positive correlation (as one variable. increases, the other increases, if one variable decreases the other also decreases)
2) Ethnographic studies
- Description: deep, lengthy qualitative analysis of a culture and its characteristics
- Strengths: provides detailed analysis and comprehensive evaluation
- Weaknesses: researcher’s presence may affect individual’s behaviour, heavily dependent on the researcher conducting the study, difficult to replicate, and objectivity may be compromised
3) Twin studies
- Description: analysis of heritability through measuring characteristics of twins
- Strengths: offers insight into how nature and nurture might interact to lead to various characteristics
- Weaknesses: Difficult to find participants who meet criteria, difficult to analyze the complex variables involved and how they interact
Heritability
the extent to which an observed trait is due to genetics versus the environment
4) Longitudinal studies
- Description: long term analysis that intermittenly measures the evolution of some behaviour or characteristic
- Strengths: scientists can understand how trait of interest changes over time
- Weaknesses: logistically demanding, expensive and difficult to implement, high attrition rate
cross-sectional study
data collection or survey of population or sample at a specific time
5) Case studies
- Description: deep analysis of a single case of the example
- Strengths: offers comprehensive details about the single case
- Weaknesses: results may not be generalizable, does not offer points of reference or comparison
6) Phenomenological studies
- Description: self observation of a phenomenon by researcher or small group of participants
- Strengths: introspection can provide insight into behaviours and occurrence that are difficult to measure
- Weaknesses: lacks objectivity due to results coming from self-analysis, difficult to generalize results to other circumstances or individuals
7) Survey
- Description: use of a series of questions to allow participants to self report behaviours or tendencies
- Strengths: easy to administer, can provide quantitative data. that can. be compared to large participant pools
- Weaknesses: self-reporting creates limitations in objectivity
8) Archival studies
- Description: analysis of historical records for insight into a phenomenon
- Strengths: provide insight into events from the past that are unique from every day behaviour
- Weaknesses: quality of analysis subject to the quality and integrity of records, difficult to conduct follow ups, data are unlikely to be comprehensive, leaving ambiguity and unanswered questions
9) Biographical studies
- Description: an exploration of all the events and circumstances of an individuals life
- Strengths: comprehensive knowledge of all the details of an individuals life
- Weaknesses: limitations in objectivity, difficult to generalize observations
10) Observational studies
- Description: a broad category that includes any research in which experimenters do not manipulate the situation or results
- Strengths: a naturalistic observation of circumstances as they are
- Weaknesses: difficult to tease out the complex interplay of many variables
Quasi-Experimental method
lacks a control group, compares same group at different time frames
Comparative method
existing groups rather than random, but does experimentally manipulate variable
common threats to external validity
experiment doesn’t reflect real world
selection criteria
situational effects
lack of statistical power
Validity
does it measure what it claims to
