Research Methods and Study Design Flashcards
1
Q
Experimental Design
A
the technical term for a specific type of research
2
Q
Steps to good experimental design
A
1) select the population
2) operationalize the independent and dependent variables
3) carefully select the control and experimental groups
4) randomly sample from the population
5) randomly assign individuals to groups
6) measure the results
7) test the hypothesis
3
Q
1) Selecting the population
A
- Objective: determine the population of interest and consider what group will be pragmatic to sample
- Common Flaws: the population is too restrictive, sampling all individuals of interest is not practical
4
Q
2) Operationalize variables
A
- Objective: determine the independent and dependent variables, specify exactly what is meant by each, make sure the dependent variable can be measured quantitatively within the parameters of the study
- Common Flaws: insufficient rigor in the description, manipulation of the independent variable presents practical problems
5
Q
Dependent Variable
A
variable that is measured
6
Q
Independent Variable
A
variable manipulated by the research team
7
Q
Operational definition
A
Specification of precisely what they mean by each variable
8
Q
Reproducibility
A
Quality of good experimental design, experiments can be reproduced by others. researchers
9
Q
Quantitative
A
numerical
10
Q
Qualitative
A
descriptive
11
Q
3) Divide into groups
A
- Objective: carefully select experimental and control groups, homogenize the two groups, isolate the treatment by controlling for potential extraneous variables
- Common Flaws: control group does not resemble treatment along important variables, the experiment is not double-blind, participants can guess the experiment allowing a placebo effect to occur
12
Q
Experimental Group
A
group of participants that receives treatment
13
Q
Control group
A
group of participants that acts as a point of reference and comparison
14
Q
Homogenous
A
a control group that is the same throughout and as similar as possible to the experimental group except for the treatment
15
Q
Extraneous (or confounding)
A
variables other than the treatment that could potentially explain the results of an experiment
16
Q
Placebo effect
A
believing that the treatment is being administered can lead to measurable results
17
Q
Double blind
A
neither the person administering. treatment nor the. participants truly know if they are assigned to the treatment or control group
18
Q
4) Random sampling
A
- Objective: make sure all members of the population are represented, ideally each member has an equal chance of being selected, meeting these criteria is often not possible for practical reasons
- Common Flaws: sampling is not truly random, sample does not represent the population of interest
19
Q
Sampling bias
A
if it is not equally likely for all members of a population to be sampled
20
Q
Selection bias
A
more general category of systemic flaws in a design that can compromise results, another example is purposefully selecting which studies to evaluate in a meta-analysis
21
Q
Meta-analysis
A
big-picture analysis of many studies to look for trends in the data
22
Q
Attrition
A
another type of selection bias, occurs when participants drop out of the study. If participants dropping out is non-random, this might introduce an extraneous variable
23
Q
5) Random assignment
A
- Objective: individuals who have been sampled are equally likely to be assigned to treatment or. control, consider matching along potential extraneous variables which have been pre-selected
- Common Flaws: groups are not properly matched, assignment is not perfectly random
24
Q
Randomized block technique
A
researchers evaluate. where participants fall along the variables they wish to equalize across experimental and control groups. Then randomly assign individuals from these groups so. that the treatment and control groups are similar along the variables of interest
25
6) Measurement
- Objective: make sure measurements are standardized, make sure instruments are reliable
- Common Flaws: tools are not precise enough to pick. up a result, instruments used for measurements are not reliable
26
Reliability
means that they produce stable and consistent results, measure what they're supposed to (construct validity) and that repeated measurements lead to similar results (replicability)
27
Psychometrics
study of how to measure psychological variables through testing
28
Response bias
another concern with surveys, defined as the tendency for respondents to not have perfect insight into their state and provide inaccurate responses
29
Between-subjects design
the comparisons are made between subjects from one group to another
30
Within-subjects design
compare the same group at different time points
31
Mixed methods research
any combination of different research techniques, such as within-subjects and between subjects, or qualitative and quantitative
32
7) Test the hypothesis
- Objective: use statistics to check for a significant difference, assign a pre-established threshold at which the null hypothesis will be rejected
- Common Flaws: small sample size leads to insufficient power, researchers do not set thresholds in advance and make after-the-fact conclusions that lead to logical fallacies
33
Type 2 error
incorrectly conclude that there is no effect (false negative)
34
Type 1 error
falsely suppose the veracity of a result that does not actually exist (false positive)
35
Null hypothesis
assume that there is no causal relationship between the variables and any effect that they measure, if there is one, is due to chance
36
Experimental hypothesis
the proposition that variations in the independent variable cause changes in the dependent variable
37
P-value
number from 0-1 that represents the probability that a difference observed in an experiment is due to chance
- lower p-values suggest a stronger relationship
38
Sample size
number of participants
39
Power
ability to pick up an effect if one is actually present
40
External validity
flaw or limitation that might make it difficult to apply conclusions to the real world
41
Internal validity
Extent to which. the outcome variable is due to the intervention. A limitation that the experiment is not "well done", leaving doubts about the conclusions because of some inherent flaw of the design.
- internal validity is high if confounding variables have been considered and minimized, and the causal relationship between independent and dependent variables can be established by the way the experiment was set up
42
Predictive validity
does the test tell us about the variable of interest
43
common threats to internal validity
```
impression management
confounding variables
lack of reliability
sampling bias
attrition effects
demand characteristics
```
44
Impression management
participants adapt their responses based on social norms of perceived researcher expectations; self fulfilling prophecy; methodology is not double-blind
Hawthorne Effect
45
Confounding Variables
extraneous variables not accounted for in the study; another variable offers an alternative explanation for results; lack of a useful control
46
Lack of Reliability
measurement tools do not measure what. they purport to, lack consistency
47
Sampling Bias
selection cirteria is not random, population used. for sample does not meet conditions for statistical test (e.g. population is not normally distributed)
48
Attrition Effects
participants fatigue; participants drop out of study
49
Demand Characteristics
participants interpret what the experiment is about and subconsciously respond in ways that are consistent with the hypothesis, respond in ways that match how they are expected to behave
50
The experiment doesn't reflect the real world
laboratory setups that don't translate to the real world, lack of generalizability
51
Selection criteria
too restrictive of inclusion/exclusion criteria for participants (i.e. sample is not representative)
52
Situational effects
presence of laboratory conditions changes outcome (e.g. pretest. and post-test, presence of experimenter, claustrophobia in an MRI machine)
53
Lack of statistical power
sample groups have high variability; sample size is too small
54
Ethical consideration
to be sure ethical standards are met, modern experiments must be cleared by an independent internal commission and contain some type of disclosure
55
Disclosure
an outline given to participants before the experiment begins that clarifies incentives and expectations while reminding them of their right to terminate the experiment at any time
56
Debriefing
participants are told after the experiment exactly what was done and why the experiment was conducted
57
Non-Experimental Designs
```
Correlational studies
ethnographic studies
twin studies
longitudinal studies
case studies
phenomenological studies
survey
archival studies
biographical studies
observational studies
```
58
1) Correlational studies
- Description: measures the quantitative relationship between two variables
- Strengths: great. preliminary technique, usually easy to conduct
- Weaknesses: does not establish causality, may not pick up non-linear relationships
59
Pearson correlation
assigns a number from -1 to +1 to a pair of variables
- if the value is negative, the two variables are negatively correlated (if one increases the other decreases and vice versa)
- positive value represents a positive correlation (as one variable. increases, the other increases, if one variable decreases the other also decreases)
60
2) Ethnographic studies
- Description: deep, lengthy qualitative analysis of a culture and its characteristics
- Strengths: provides detailed analysis and comprehensive evaluation
- Weaknesses: researcher's presence may affect individual's behaviour, heavily dependent on the researcher conducting the study, difficult to replicate, and objectivity may be compromised
61
3) Twin studies
- Description: analysis of heritability through measuring characteristics of twins
- Strengths: offers insight into how nature and nurture might interact to lead to various characteristics
- Weaknesses: Difficult to find participants who meet criteria, difficult to analyze the complex variables involved and how they interact
62
Heritability
the extent to which an observed trait is due to genetics versus the environment
63
4) Longitudinal studies
- Description: long term analysis that intermittenly measures the evolution of some behaviour or characteristic
- Strengths: scientists can understand how trait of interest changes over time
- Weaknesses: logistically demanding, expensive and difficult to implement, high attrition rate
64
cross-sectional study
data collection or survey of population or sample at a specific time
65
5) Case studies
- Description: deep analysis of a single case of the example
- Strengths: offers comprehensive details about the single case
- Weaknesses: results may not be generalizable, does not offer points of reference or comparison
66
6) Phenomenological studies
- Description: self observation of a phenomenon by researcher or small group of participants
- Strengths: introspection can provide insight into behaviours and occurrence that are difficult to measure
- Weaknesses: lacks objectivity due to results coming from self-analysis, difficult to generalize results to other circumstances or individuals
67
7) Survey
- Description: use of a series of questions to allow participants to self report behaviours or tendencies
- Strengths: easy to administer, can provide quantitative data. that can. be compared to large participant pools
- Weaknesses: self-reporting creates limitations in objectivity
68
8) Archival studies
- Description: analysis of historical records for insight into a phenomenon
- Strengths: provide insight into events from the past that are unique from every day behaviour
- Weaknesses: quality of analysis subject to the quality and integrity of records, difficult to conduct follow ups, data are unlikely to be comprehensive, leaving ambiguity and unanswered questions
69
9) Biographical studies
- Description: an exploration of all the events and circumstances of an individuals life
- Strengths: comprehensive knowledge of all the details of an individuals life
- Weaknesses: limitations in objectivity, difficult to generalize observations
70
10) Observational studies
- Description: a broad category that includes any research in which experimenters do not manipulate the situation or results
- Strengths: a naturalistic observation of circumstances as they are
- Weaknesses: difficult to tease out the complex interplay of many variables
71
Quasi-Experimental method
lacks a control group, compares same group at different time frames
72
Comparative method
existing groups rather than random, but does experimentally manipulate variable
73
common threats to external validity
experiment doesn't reflect real world
selection criteria
situational effects
lack of statistical power
74
Validity
does it measure what it claims to
