Week 6.1: Research Design Flashcards
Key questions for quan. research design interventions
will there be an intervention?
what types of comparisons will be made?
How will confounding variables be controlled?
will blinding be used?
how often will data be collected?
when will “Effects” be measured, relative to potential causes?
The key goal for quan research design is
to make the best possible design for a study possible that controls for many possible weaknesses but can work with our question
Key Design Features that determine research design options
Intervention
Comparison
Control over confounding variables
Blinding
Time frames
Relative timing
Location
Affordability
There is no perfect..
research study
Many (if not most) quan research questions are about ___ and ___ (___)
cause; effects (causality)
Research questions that seek to illuminate causal relationships need to…
be addressed with appropriate design
*every research deign has its strength and the goal is to get to the next level of research and work up to a causal study
Counterfactual
it is what would have happened to the same people exposed to a “Cause” if they simultaneously were NOT exposed to the cause
basically the control - what if they did not get the condition of interest
Counterfactuals are harder to get when
we cannot control the intervention!
What is this: What would the COVID-19 rate be like if NOT for vaccination
counterfactual
It is harder to prove counterfactuals in areas like…
public health - hard to prove a negative (if prevention works then its hard to prove)
counterfactuals are more hypothetical in population research
Effect
represents the difference between what actually did happen when exposed to the cause and what would happen with the counterfactual condition
What are the 3 criteria for causalit/making causal inferences
Temporal
Relationship
Cofounder
Temporal Aspect of Causal Inferences
the idea that the cause MUST PRECEDE THE EFFECT in time (not the other way)
Relationship Aspect of Causal Inferences
the idea that there MUST BE A DEMONSTRATED ASSOCIATION between the cause and the effect
Cofounder Aspect of Causal Inferences
the idea that the relationship between the presumed cause and effect CANNOT BE EXPLAINED BY A THIRD VARIABLE OR COFOUNDER
another factor related to both the presumed cause and effect cannot be the “real” one
ex: “people drinking a lot of coffee have higher rates of lung cancer” = confoundvariable is that those drinking coffee also smoked cigarettes
What is the additional fourth criteria for causal inferences in research when its based in health research/nurse research
Biological Plausibility
Biological Plausibility
A fourth criteria for causality needed by health research
it states the causal relationship should be consistent with evidence from basic physiologic studies
this stops use from grabbing two things and making magical conclusions
What are the 4 types of questions of health research that determines the different designs used for research
- Therapy Questions
- Prognosis Questions
- Etiology/Harm Questions
- Description
What design offers the strongest evidence of whether a cause (intervention) results in an effect (a desired outcome)
Experimental Designs (RCTs)
Thats why they are high on evidence hierarchies for questions about cause and effect
What makes a quasi experimental design different
you cannot randomize the sample
What makes a cohort study, case, or descriptive/correlational design different
cannot manipulate the IV in these cases
Hierarchy of Research Design in Therapy Questions
- RCT/Expt Design
- Quasi design
- Cohort study
- Case control
- descriptive/correlational
Hierarchy of Research Design in Prognosis Questions
- Cohort Study
- Case Control
- Descriptive/Correlational
RCT is not available so you have to start lower which in this case is a prospective study that looks retrospectively
Hierarchy of Research Design in Harm Questions
- RCT/Experimental
- Quasi Experimental
- Cohort Study
- Case Control
- Descriptive/correlational
We control things again in this type of question so we can now use RCT and quasi once more
You can use what kind of sampling compairsons in a research study design
within groups
between groups
both
Within Groups Comparison
comparisons about measurements made with the same subjects at different points in time
Between Groups Comparison
Comparisons made with more than one group of subjects at one or more points in time
Schema
shorthand visual description of the design of a study
What does O, X, and R mean in a schema
R = its an experimental design/randomization
O = observation
X = intervention
What would O1 X O2 mean
it means Group one observed at one time and then group one observed at a time after an intervention
Which groups are we interested in comparing in the schema:
O1 X O2
O3 O4
O2 and O4 because they are outcome and control post interventions
Intervention
the researcher doing something to some subjects - introducing an intervention or treatment
pre and post tests
Control
the researcher introduces controls, including the use of a control and experimental groups
can be its own group or a group acting as its own control
Randomization
the experimenter assigns participants to a control or experimental condition on a random basis
the purpose is to make the groups equal with regard to all other factors except receipt of the intervention
You always need what 3 things for an experimental design schema
Intervention (X)
Randomization (R)
Control
Outcome (O)
The symbol O means
measurement at a point in time
observation, data collection
The symbol X means
intervention or treatment
sometimes listed as T
The symbol R means
randomization
If you can manipulate the IV and you can randomize what sort of research design can be used
EXPERIMENTAL DESIGN:
“Classic” Experiment
Post Test Only
Solomon 4 Group
Crossover
A quasi experimental study lacks what part of the schema
R
Posttest-Only (or after only) Experimental Design
outcome data collected only after the intervention
Schema:
R X O
R O
Why is posttest-only design risky experimental design
because you do not know a baseline
Pretest-posttest (before-after) design
outcome data collected both at BASELINE and after the intervention
Schema:
R O X O
R O O
An RCT / Classic Experimental Design
Crossover Design
Subjects are exposed to 2+ conditions in random order and the subjects serve as their own control
Schema:
R O Xa O Xb O
R O Xb O Xa O
Why is crossover design risky experimental design
because if the participant favors one intervention over another they may continue using it even when you dont want them too
this is one of the only ways you will ever see a one group experimental trial
Protocol
the way someone describes their intervention in a study
should be replicable only by reading
“Operational Definition” of the IV
The sample serves as a proxy for what? The results serve as a proxy for what?
general population; the intervention
The more complex an intervention…
the more the results may diverge from real life
Intervention Fidelity
Treatment Fidelity
Whether the treatment as planned was actually delivered and received the way that it was supposed to
ex: Even at home medicaitons are a proxy for what is ideal as things like inaffordability, dislike, etc all mess with fidelity and effect if it transfers to real world situations
“Usual Care”
standard or normal procedures used to treat patient
the control group would get this, and the experimental group may get this on top of an intervention for safety and ethic reasons (beneficence)
Placebo
pseudointervention
when a fake intervention with no presumed therapeutic value is used
Delayed treatment and the Attention-control condition
it means wait listing the control group for the intervention
if it is of benefit you must give it to the control group
Attention Control
control group condition
extra attention but not the active ingredient of the intervention
so it is the researcher giving extra attention to you to make it appear you are not just in the control group
Delayed Treatment (“Wait Listed Controls”) Control
the intervention is given at a later date - often for measurement at baseline adn then look at outcome of the experimental gorup and then give it to the control if ok and after
Schema of a Delayed Treatment Control
R O X O O
R O O X O
Advantages of Experiments
most powerufl for detecting cause and effect relationships
Disadvantages of Experiments
often no feasible or ethical, Hawthrone effect (knowledge of being in a study may cause people to change their behavior), often expensive
Quasi Experiments
involve an intervention but LACK either:
randomization
or
control group
2 categories of quasi experimental designs
- nonequivalen control group designs
- within subjects designs
Non equivalent control group designs
quasi experimental
those getting the intervention are compared with a nonrandomized comparison group
Within subjects designs
one group is studied before and after the intervention
quasi experimental
What design is it if you can manipulate the IV but you cannot randomize
quasi experimental:
one group pre test/post test
non equivalent control group; pre test/posttest
one group time series
pre experimental
non equivalent control group post test only
1 group post test only
One Group Time series
quasi experimental
measure repeatedly and often
the intervention is a rule o policy change and you continuously measure to see for change
O1O2XO3O4
Nonequivalent Control group pretest posttest design
If pre intervention data are gathered, then the comparability of the experimental and comparison groups at the start of the study can be examined
quasi experimental: “Non equivalent control group pretest posttest design” - a non randomized or it doesnt have a control group
O1 X O2
O3 O4
Nonequivalent control group posttest only
quasi experimental
very weak
withou pre intervention data it is risky to assume the groups were similar at the outset
X O1
O2
Within Subjects Experiment: One group pretest-posttest designs
quasi experiment
typically yield extremely weak evidence of causal relationships
O1 X O2
Within Subjects Experiment: Time Series Design
quasi experiment
gather a preintervention and postintervention data over a longer period
usually go and work at the population level by looking at rates - vaccine status over time, smoking over time, car crashes before and after seatbelt mandate
O1O2O3O4XO5O6O7O8
Advantage and Disadvantage of Quasi Experiments
May be easier and more practical than true experiments BUT
makes it more difficult to infer causality and usually there are several alternative RIVAL HYPOTHESES for results
Non experimental studies
if reserachers do not intervene by controlling IV, the study is nonexperimental (observational)
still quantitative
Why use non experimental design
not all IV (Causes) of interest to nurse researchers can be experimentally manipulated
ex: gender or smoking - not ethical or practical
Ex Post Factor Non Experimental Design
Taking one observation/measurement at a set time
schema:
O1
Most non experimental studies are ____ design
correlational
Correlational Designs
Cause probing questions (prognosis, etiology/harm, etc) for which manipulation isnt possible are typically addressed with correlational design
weaker than RCTs for cause probing questions, but different designs offer varying degrees of supportive evidence
weaker than causality because we dont know if intervention/cause lead to result or vice versa
Correlation
an association between variables and can be detected through statistical analysis
Prospective Correlational Design
non experimental
A potential cause in the present (ex: Experiencing v not experiencing a miscarriage) is linked to a hypothesized later outcome (ex: depression 6 mo later)
Another name for Prospective Correlational Design is
Cohort Study
What is stronger, prospective correlational design or retrospective correlational design
prospective - it is a little better at supporting causal influences because of the choice to ata to collect as compared to holding onto what data was drawn and having to stick to it
still not as strong as experimental design
Retrospective Correlational Design
non experimental
outcome in the present (ex: depression) is linked to a hypothesized cause occurring in the past (ex: having had a miscarriage)
Another type of Restrospective design is…
case control design!
Where cases (ex: those with lung cancer) are compared to controls (ex: those without) on prior potential causes (e: smoking habits)
Descriptive Research
Not all research is probing like in experimental, quasi, and non experimental
the purpose of this is to observe, describe, and document aspects of a situation - NOT QUALITATIVE, a type of non experimental design
some research is descriptive (ex: ascertaining the prevalence of a health problem like COVID-19) or descriptive correlational (where the purpose is to describe whether variables are related, without ascribing a cause and effect connection at all)
Advantage of nonexperimental research
efficient way to collect large amounts of data when intervention and/or randomization is not possible
can collect a lot of data and acts as a good place to start climbing the research design ladder
Disadvantages of nonexperimental research design
does no yield persuasive evidence for casal inferences
this is not a problem when the aim is described though, but correlational studies are often undertaken to discover causes
Time Dimensions in Research Design
Cross Sectional Design
Longitudinal Design
Cross Sectional Design
data is collected at a single point in time
Longitudinal Design
data is collected two or more times over an extended period
beneficial as it is better at showing patterns of change and at clarifying whether a cause occurred before an effect (outcome)
What is the challenge despite the benefit of longitudinal design
Attrition
Attrition
loss of participants over time
Ways to control the study context/external factors
- Achieve constancy of conditions
- Control over environment, setting, time
- Control over intervention via a formal protocol: intervention fidelity
we can control thigns v well but the flip side is the more we control the less people actually act like that irl
Ways tot control participant factors
- Randomization (or subjects as own controls (crossover design))
- Homogeneity (restricting the sample) - but may lessen generalizability
- Matching
- Statistical Control - ex: ANCOVA
What is Matching
taking demographics you want an making sure the control and experimental group are demographically similar
Characteristics of Good Quantitative Research Design
High statistical conclusion validity
High internal validity
High external validity
Construct validity
Validity
degree to which you can trust the results of your research study
Statistical Conclusion Validity
the ability to detect true relationships statistically
Internal Validity
the extent to which it can be inferred that the IV caused or influenced the DV and not other factors
External Validity
the generalizability of the observed relationships across samples, settings or time
Construct Validity
the degree to which key constructs are adequately captured in the study
how well we defined what is being studied
Threats to Statistical Conclusion Validity
low stat power (small sample size - big one!)
weakly defined “cause” - IV not powerful
unreliable implementation of a treatment - low intervention fidelity
if group is not a good proxy to population of interest
What is the relationship of internal and external validity
its like a see saw
as things get more tightly controlled for research internal validity rises but external validity (generalizability) decreases
Threats to Internal Validity
Temporal Ambiguity
Selection Threat
History Threat
Maturation threat
Mortality and Attrition threat
Temporal Ambiguity Threat
internal validity threat
to establish causation you need an event to proceed the effect
Selection Threat
biases arising from preexisting differences between groups being compared
threat to int validity
does it actually apply to the group you hope to generalize to
What is the single biggest threat to internal validity
selection threat
History threat
other events co occurring with causal factor that could also affect outcomes
threat to int validity
ex: NUrsing school stuck online because of pandemic, changes NCLEX pass rates
Maturation Threat
processes that result simply from the passage of time
ex: disease getting worse over time, children aging
threat to in validity
Mortality/Attrition threat
diffential loss of participants from different groups
typically a threat in experimental studies
threat to int validity
When does attrition become a big issue for a study
When the groups are no longer equal at baseline - this is a key reason to ahve baseline measurements with 2 groups, so you can reevaluate and check even if some people are lost
How to reduce history threat
random selection or assignment
*that way changes are seen equally in both groups
How to reduce maturation threat
control groups, random assignment, and tkaing baseline data
We need to see that maturation impact in both groups to control baseline changes
How to reduce selection threat
random sampling and random assignment
MOST COMMON THREAT
we may only have access to convenience sampling so we need to use probability assessment methods and random assignments so they are similar to one another by group equally
How to reduce mortality/attrition threat
collection of demographic variables from all subjects for future comparison to those who complete
communicate clear instructions and expectations regarding participation in the study from the start
A 2 FOLD WAY TO STOP THIS
Attrition of ___% or lower is acceptable
20%
Instrumentation threat
threat to in validity
comes from inconsistency in data collection - so the way we collect data affects type of data we get (ex; an anon questionnaite may get different answers than an in person interview)
How to reduce instrumentation threat
comprehensive trainin of data collectors
reliability and validity of instruments
Anything that changes the outcome that is not the IV is…
a threat to internal validity
Testing Threat
threat to int validity
Multiple testing might influence subjects response on subsequent testing just the act of testing and repeating may influence results
How to reduce Testing Threat
Post test only design
Control group tested the same # of times or prolong length of time between the tests
What are the 3 threats to external validity
person
time
place
How does person, time and place threaten external validity
Homogenous samples (person) increasing internal validity will threaten generalizability
Healthcare or experiments in other countries or areas can really impact replication in other areas like the USA in many ways (Place)
technological interventions like obseletion can threaten replication and generalizability (Time)
Construct Validity
general question of if the intervention is a good representation of the underlying construct
Threats to construct validity
is the intervention a good representation of the underlying construct
is the intervention or awareness of the intervention that resulted in benefits
does the DV really measure the intended construct
Basically, is this a good PROXY FOR WHAT WE ARE TALKING ABOUT IRL
T/F: An experimental research design involves a nonrandomized controlled trial
False
*A quasi experimental research design involves a controlled trial without randomization
Which characteristic is a key criterion for causality?
A. cause occurring before the effect
B. third variable involved with the cause and effect
C. no empirical relationship between the cause and effect
D. single source evidence about the relationship
A. cause occurring before the effect
*three key criteria for causality are cause preceding effect in time, demonstrated empirial relationship between the cause and effect, relationship not being explained by a third variable - an additional criterion is that evidence of the relationship should come from multiple resources
T/F: A true experiment requires that the research manipulate the IV by administering an experimental treatment (Or intervention) to some subjects while withholding it from others
B. True
*In a true experiment the reseracher manipulates or does something, usually an intervention or treatment, to some subjects and not to others
Which design is considered a quasi experimental research design?
A. pretest posttest design
B. posttest only design
C. crossover design
D. within subjects design
D. within subjects design
*Quasi experimental research designs include non equivalent control groups and within subject designs - the other designs are used for expt research
T/F: Cross sectional research designs are helpful in showing patterns of change
False
*Longitudinal studies in which data are colelcted two or more times over an extended period, are better at showing patterns of change than cross sectional studies, which collect data at a single point in time
