⚖️430: Research Methods QUANTITATIVE FINAL Flashcards
What are the major classes of quantitative design?
- Experimental (and Quasi-experimental)
2. Non-experimental
What are the 3 criteria of causality?
- Preceded the effect in time
- Association between the cause and effect
- Relationship cannot be due to the influence of a third variable or confounder
What are the 3 aspects of experimental design?
- Manipulation
- Control/comparison
- Randomization
What are the different experimental designs?
- Randomized controlled trial (POSTTEST ONLY)
- Randomized controlled trial (PRETEST-POSTTEST)
- Cross-Over Design
What is a cross-over experimental design?
Sample:
- Treatment 1 ➡️ washout ➡️ Treatment 2
- Treatment 2 ➡️ washout ➡️ Treatment 1
What is a pretest-posttest randomized controlled trial experimental design?
Measures outcomes before and after experimental and control interventions
What are limitations of experimental designs?
- Not everything can be manipulated
- Hawthorne Effect
- Blinding not always possible
- May be unethical to withhold care
What are quasi-experimental designs?
Involves a manipulation but lacks either randomization or control group
What are the 2 categories of quasi-experimental designs?
- Non-equivalent control group design
(Intervention group compared to nonrandomized control group) - Within-subjects designs
(One group studied before and after intervention)
What are strengths of quasi-experimental designs?
More feasible as compared to a true experiment
What are limitations of quasi-experimental designs?
- More difficult to infer causality
2. Rival explanations for results
What are categories of non-experimental designs?
- Correlational cause-probing research
- Descriptive correlational designs
- Univariate descriptive studies
- Cohort studies
- Case-control study
What is a cohort study?
Investigator identified exposed and none posed groups (cohorts) and follows them forward in time.
- Useful when harmful outcomes occur infrequently
***Exposed and unexposed may begin with different risks of target outcomes (CONFOUNDING)
What are strengths of non-experimental designs?
- Efficient way to collect large amounts of data when intervention/randomization no possible
- Does not require artificial provision of exposure
- Treatments not withheld
What are limitations of non-experimental designs?
- Rival explanations for results
2. Limited ability to infer causality
Cross-sectional
Data collection at one time point, or more than one in close succession
Longitudinal
- Data collected at multiple time points over days/months/years
- Better at showing patterns of changed and at clarifying whether a chase occurred before an effect
- ATTRITION = loss of participants over time
Which is NOT another term for randomization? A. Random sampling B. Random allocation C. Random assignment D. None of the above
A. Random sampling
What are the 4 main aspects of Validity?
- Statistical Conclusion Validity
- Internal Validity
- Construct Validity
- External Validity
Validity
The degree to which inferences made in a study are accurate and well-founded
What are ways of controlling extraneous/confounding variables?
- Constancy of conditions
- Formal protocol to enhance intervention fidelity
- Randomization
- Homogeneity (restricting sample)
- Matching
- Statistical control (ex. Analysis of covariance)
Statistical conclusion validity
The ability to detect true relationships statistically
What are threats to statistical conclusion validity?
- Low statistical power (ex. low sample size)
- Weakly defined “cause” - independent variable poorly constructed
- Low implementation fidelity
- Poor intervention adherence
Internal validity
Extent to which it can be inferred that the independent variable caused or influenced the dependent variable
What are threats to internal validity?
- Temporal ambiguity
- Selection threat - bias arising from preexisting differences between groups
- History - other events co-ocurring with the causal factor
- Maturation - processes that result from passage of time
- Mortality/attrition - differential loss from groups
Construct validity
Key constructs are adequately captured in the study and thus the evidence in a study supports inferences about the constructs that are intended to represent.
What are threats to construct validity?
- Poor construct validity of measurement tools
- Reactivity to the study situation (Hawthorne Effect)
- Researcher expectancies
- Novelty effect
- Compensatory effects
- Treatment diffusion or contamination
External validity
The extent to which it can be inferred that the relationships observed in a study hold true in other samples or settings (ie. generalizability)
What are threats to external validity?
- Sample that is mom-representative of population
- Intervention that is difficult to replicate
- Artificiality of research environment
Open-ended questions
- Allows for more in-depth data
2. Analysis can be difficult and time-consuming
Closed-ended questions
- Greater privacy
2. Less likely to go unanswered
Composite psychosocial scales
Used to make fine quantitative discriminations among people
With different attitudes, perceptions, or needs
Likert Scales
- Consist of several declarative statements (items) expressing viewpoints
- Responses are on ah agree/disagree continuum (usually 5 to 7)
- Responses to items are summed to compute a total scale score (SUMMATED RATING SCALE)
Observation in quantitative studies
- Structured observations of pre-specifies units (ex. Behaviours, actions, events)
- Structures in what to observe, how long, and how to record
Methods:
- Category systems
- Checklists
Bromage Score
Observational rating on a descriptive continuum to test degree of motor block after epidural
What are disadvantages of observations?
- REACTIVITY = Behaviours May be altered by awareness of being observed
- Observer bias
- Resources required to gain entry into setting
In vivo measurements
Performed directly within or on living organisms (ex. Blood pressure)
In vitro measurements
Performed outside the organism’s body (ex. Urinalysis)
What are advantages of biophysical measures?
- Precision
- Objective
- Validity
What are disadvantages of biophysical measures?
- Resources required
- Many factors may affect variability
- Ethical responsibilities
What are the 2 types of biophysical measures?
- In vivo
2. In vitro
Errors of Measurement
Obtained score = True score +/- Error
= Signal +/- Noise
Reliability
The extent to which scores are free from measurement errors.
Reliability is consistency of measure, validity is accuracy of measure.
Reliability coefficients
0.00 to 1.00
Unsatisfactory < 0.70
Desirable >= 0.80
What are the 3 methods of assessing reliability?
- Test-Retest Reliability
- Internal Consistency
- Interrater Reliability
Test-Retest Reliability
Administration of the same measure to the same people on two occasions. Check correlation of test scores with retest scores.
Internal Consistency
(Cronbach’s a)
Consistency across items in a composite scale. Instrument is administered on one occasion and the relationship between items tested.
Interrater Reliability
Similarity between measurements of multiple observers/raters using the same instrument.
Instrument Validity
The degree to which an instrument measures what it is supposed to measure
What are the 4 aspects of instrument validity?
- Face validity
- Content validity
- Criterion-related validity
- Construct validity
Face validity
Refers to whether the instrument looks as though it is measuring the appropriate construct. Based on judgment, no objective criteria for assessment.
Content Validity
The degree to which an instrument has an appropriate sample of items for the construct being measured.
Content Validity Index (CVI)
- Expert evaluation
- Quantitative measure
- Proportion of items that are highly relevant on a numerical scale
Desired >= 0.8
Criterion-Related Validity
The degree to which scores on an instrument are a good reflection of a “gold standard” criterion for the same construct.
How is Criterion-related Validity evaluated?
- Concurrent Validity = correlated with external criterion, measured at the same time
- Predictive Validity = correlates with external criterion, measured at a future point in time
Construct validity
Degree evidence captures the construct of interest. Especially useful for when there is no “gold standard” comparison.
How is construct validity evaluated?
Known-groups (Discrimination) Validity
- Convergent Validity = test correlates with other measures of same construct
- Divergent Validity = test DOES NOT correlate with measures of other/different constructs
How are measurements have both Reliability and Validity?
Reliability is necessary but not sufficient for validity.
Together the Obtained Score = True Score
Statistical Significance
The results from the sample data are unlikely to have been caused by chance.
Clinical Significance
The results have practical importance
Confidence Intervals (CI)
Range of values within which a population parameter is expected to lie.
The narrower the CI, the more precise the estimate of effect.
Relative Risk vs Absolute Risk
Absolute = likelihood event will occur under specific conditions
Relative = likelihood event with occur in a group compared to another group with different behaviours, environments, etc
Clinical significance of a given Relative Risk cannot be made unless you know the Absolute Risk.
Number Needed to Treat (NNT)
Number of people that would need to receive the intervention to prevent one additional bad outcome.
Lower baseline absolute risk ➡️ higher NNT
Higher baseline absolute risk ➡️ lower NNT
What are the 5 types of reviews?
- Systematic Review
- Meta-Analysis
- Meta-Synthesis
- Scoping Review
- Narrative Review
What are the 6 steps in conducting a Review?
- Research Question
- Sampling of Primary Studies
- Quality Appraisal of Primary Studies
- Data extraction
- Data analysis
- Evidence Synthesis
Systematic Review
Rigorous synthesis of research findings in a particular research question, using systematic sampling and data collection procedures and a formal protocol.
Goals:
- Reduction of bias and random error
- Transparency
- Reproducibility and Verifiability
Literature search in a systematic review
- Detailed and Exhaustive
- Search strategy shoul reflect focused clinical question
- High yield expected
Data extraction in a systematic review
- Collect relevant study information
2. Preferably performed in duplicate
Analysis in a systematic review
- Qualitative synthesis (Narrative description) = provides overview of the available studies, common results, discrepancies, etc
- Quantitative synthesis (Meta-analysis) = only possible if studies are similar enough to be combined statistically
Meta-Analysis
- Statistical process for combining results of multiple studies
- Attempts to overcome the problem of reduced statistical power in small sample sizes by combining results
- Results often depicted in a Forest Plot (or Blobbogram)
Metasynthesis
Integration and/or comparison of findings from multiple qualitative studies.
Purpose = generate new knowledge
Scoping Review
Determines the general state of knowledge related to a specific question and locate gaps in the literature.
Broader scope that a systematic review, but follows an established methodology
Narrative Review
- Depends on authors’ bias
- Author picks criteria
- Search any databases, less structures and comprehensive
- Methods not usually specified
- Only narrative summary
- Can’t replicate review
Systematic Review
- Scientific approach to a review article
- Criteria determines at outset (a priori)
- Comprehensive systematic search for relevant articles; use of systematic strategy
- Explicit methods of appraisal and synthesis
- Meta-analysis
- Reproducible
Clinical practice guidelines development process
- Establish guideline development group
- Develop practice recommendations
- External review
- Monitoring and Updating
What are the 6 Levels of Evidence?
Ia. Meta-analysis or Systematic reviews of randomized controlled trials
Ib. At least one randomized controlled trial
IIa. At least one well-designed controlled study without randomization
IIb. At least one other type of well-designed quasi-experimental study without randomization
III. Well-designed non-experimental descriptive studies, such as comparative or correlation
IV. Expert committee reports or opinions
What are types of clinical practice guideline developers?
- Government agencies (ie. Ontario)
- Professional Associations (ie. RNAO)
- Disease or Population-Specific Organizations (ie. Heart&Stroke)
- International Organizations (ie. WHO)
- Other Organizations (ie. CAMH)
What are challenges with Knowledge Transmission (KT)?
- Using plethora of terms to describe KT
- KT engages researchers and knowledge users with different perspectives
- KT involves a complex set of interactions
- Successful KT involves practice chance to change outcomes
Ethical imperative for Knowledge Transmission (KT)
- (1/3) of patients DO NOT get treatments of proven effectiveness
- (3/4) of patients do not have info they need for decision making
- (1/4) of patients get care that is NOT NEEDED or Potentially harmful
- (1/2) of physicians DO NOT have evidence they need for decision making
Diffusion of innovations (Rogers,1962,2003)
Influential theory that generally assumed to represent or form theoretical foundation of research utilization or KT. Explains the process and spread of an innovation within a social system.
What are the 4 main factors that influence diffusion?
- Innovation
- Communication (method to inform others)
- Time (from first knowledge to acceptance/rejection)
- Social system
What are the 5 steps in the process of innovation adoption?
- Knowledge
- Persuasion
- Decision
- Implementation
- Confirmation
PEST Analysis
Societal Characteristics
- Political
- Economic
- Socio Cultural
- Technological
KT (implementation) Strategies
- Reminders
- Educational (written) Materials
- Educational Outreach
- Audit & Feedback
What is a Tailored Intervention?
Interventions planned after investigating factors that explain current practice and reasons for resistance to change.
- Barriers identified through observation, focus groups, interview, surveys
A theoretical integration of qualitative findings is known as a:
Meta-synthesis
Can purposive sampling be used in quantitative research?
Yes
What is a Type I statistical error?
The rejection of a null hypothesis when it should not be rejected
(False positive)
Risk is controlled by the level of significance (alpha) that is < 0.5 or 0.1
Type II Statistical Error
Failure to reject the null hypothesis when it should be rejected.
False negative.
Controlled by ensuring adequate power.
Can a low response rate threaten the external validity of quantitative research study?
Yes
What are the 4 levels of Measurement?
- Nominal = data classified into categories (ie. gender, diagnosis)
- Ordinal = ranked categories (ie. disease stage)
- Interval = meaningful difference between values (ie. temperature, shoe size)
- Ratio = Meaningful difference between values within absolute zero (ie. height, fatigue score)
Descriptive Statistics
Used to present, organize, and summarize the data from the sample.
- Univariate statistics
- Bivariate statistics
Inferential Statistics
Used to make inferences about the population from the sample
Univariate Descriptive Statistics
Measures of Central Tendency:
- Mean
- Median
- Mode
Measures of Dispersion:
- Range
- Standard Deviation (SD)
- Variance
Positively skewed distribution
/\__
Negatively skewed distribution
___/\
Contingency Table - Bivariate descriptive statistics
A two-dimensional frequency distribution; frequencies of two variables are cross-tabulated.
Cells at intersections of rows and columns display counts and percentages.
Variables Nominal or Ordinal
Correlation = Bivariate descriptive statistics
Indicates direction and magnitude of relationship between two variables.
Interval-ratio measures.
Correlation coefficient = Pearson’s r
Null Hypothesis
Statistical tests to either reject or fail to reject the null hypothesis.
H0: There is no difference/relationship between the IV and DV.
H1: There is a difference.
P<0.05
There is less than 5% chance that the difference is observed given the null hypothesis.
REJECT the null hypothesis
P>0.05
There is a greater than 5% chance that the difference is observed given the null hypothesis.
FAIL TO REJECT the null hypothesis.
Statistical Power
A measure of the probability that the statistical test will detect significant difference/effect IF ONE EXISTS.
How is statistical power determined?
- Effect size (size of treatment/relationship effect)
- Alpha level
- Sample size
Smaller effect size = lower power
Smaller alpha = lower power
Power analysis should be done in advance to determine sample size.
Parametric Statistics
For interval-ratio data that are normally distributed.
Ex:
- T-tests
- ANOVA
- Correlation
- Regression
Non-Parametric Statistics
For nominal-ordinal data or non-normally distributed interval-ratio data.
Examples:
- Chi-squared (x2) Test
- Mann Whitney U test
What is a 95% Confidence Interval?
Confident that the true point estimate lies within this range 95 times out of 100.
How do you determine differences in continuous outcomes?
A confidence interval that includes 0 (ie. -0.5 - 4.8) signifies that it is plausible that there is no difference.
How do you determine differences in odds ratios and relative risk?
A confidence interval that includes 1 (ie. 0.1-3.2) signifies that it is plausible that there is no difference.
Cronbach’s alpha
The mean of all possible split-half correlations. Measures internal consistency.
Convergent validity
A form of criterion validity where the criteria includes other measures of the same construct
Relative Risk
Ratio of probabilities comparing risk of event among those exposed and not exposed.
Example:
(Risk of event in Treatment Group)/(Risk of event in control group)
Odds ratio
Compares the presence to absence of an exposure given already known outcome.
Example:
(Odds of event in Treatment group)/(Odds of event in control group)
Triangulation in Quantitative Research
Using multiple measures of an outcome variable to see if predicted effects are consistent.
Ex: Sleep diary + Actigraphy + Polysomnography
