Unit 3 Flashcards
what is validity
degree to which the study establishes relationships it claims to establish
Trust
By determining quality of the study design you assess validity of study. why
because you trust results of studies with higher quality study design than studies with lower quality design
We tend to trust the findings from a study if:
the study design is of high-quality & is valid
The degree to which the study establishes the relationship that it claims to establish is more formally referred to as:
study design validity
The three primary types of validity that make up the overall validity of a study design include:
statistical validity
internal validity
external validity
what is statistical validity
concerned w/ whether appropriate statistical procedures were used to determine sample size & analyze data
Using Inappropriate Statistical procedures for data analysis leads to
Invalid conclusions about data & relationship bw IV & DV
In order to use a specific statistical test for analysis, the study design and data must meet very specific requirements and assumptions.
t
The misuse of a statistical test can misrepresent the relationship between the independent variable and dependent variables, leading to poor statistical validity.
t
what is internal validity
degree to which the change in the DV is caused by the manipulation of the IV
whether the IV and only the IV caused a differential change in the DV across groups
One group pre-test post-test design
follow 1 group of PTs over 2 time points
what are two threats to IV
maturation & hx effects
Potential explanations for change in DV
threats to IV
waht are maturation effects
natural changes over time in DV regardless of intervention
what are hx effects
confounding factors (extraneous variables) that are introduced between pre and post-test measurements
loss of participants during the study
attrition threats
when participants are tested multiple times & leads to unclear results if the change is due to an intervention or them being tested so many times
testing threats
changes in the measurement tools or procedures influence the outcome and/or measurement of DV
instrumentation threats
statistical phenomenon where is something is measured more the once the scores on subsequent measurements tend to be closer to the mean than initial measurement
If participants are selected for a study based on extreme scores (e.g., very low or very high), their scores on subsequent measurements might move closer to the mean not because of an intervention, but simply due to this statistical phenomenon.
regression to the mean threats
systemic differences between or within groups at the ouset of the study
Occurs when participants are not randomly assigned to groups
Ex: one group is older, healthier and more motivated than the other
Differences could be due to initial differences in the groups rather than intervention
selection threats
Refers to natural change over time of a variable
maturation
maturation
Refers to natural change over time of a variable
Confounding factors that are introduced to the study between measurement time points
hx effects
hx effects
Confounding factors that are introduced to the study between measurement time points
Term used to describe alternative explanations for changes in the DV
threats to IV
Change in the DV is due to the manipulation of the IV
internal validity
describe internal validity
Change in the DV is due to the manipulation of the IV
what is external validity
degree to which the results of study can be generalized to the larger target population
Concerned w/ usefulness of the results in the real clinical world
Can we replicate study findings in our own routine practice
what are the threats to external validity
do the participants look like PTs we care for? And does the study setting look like the PT care setting?
what are efficacy studies
studies occurring in well-controlled environments with optimal treatment conditions
studies occurring in well-controlled environments with optimal treatment conditions
efficacy studies
studies occurring during routine patient care and under typical treatment conditions
effectivenesss studies
what are effectiveness studies
studies occurring during routine patient care and under typical treatment conditions
When assessing the external validity of a study, you must consider the interaction between treatment and selection. This threat is most concerned about:
the study sample
In general, as you increase internal validity, external validity will:
decrease
When critically reviewing a study determine first
the research design that was used & extraneous variables
what are extraneous variables
factors not directly related to the purpose of the study but affect results of the study (DV)
often confounding and bias
factors not directly related to the purpose of the study but affect results of the study (DV)
often confounding and bias
extraneous variables
what are the two types of extraneous variables
instrinsic & extrinsic factors
what are intrinsic factors
variables representing personal characteristics or attributes of the participant (maturation, healing, aging)
variables representing personal characteristics or attributes of the participant (maturation, healing, aging)
intrinsic factors
what are extrinsic factors
variables from the environment and situation (changes in rules & regs and societal attitudes)
variables from the environment and situation (changes in rules & regs and societal attitudes)
extrinsic factors
Actively do these when reviewing articles
Identify the extraneous variables that can impact results & determine if they have appropriately controlled/accounted for extraneous variables
These provide best demonstration of cause and effect relationships bw IV & DVs
experimental designs
types of experimental designs
Randomized controlled trials - aka true experimental designs
Quasi-experimental designs
Single-subject designs (or n-of-1 designs)
Factors associated with personal characteristics or attributes of a participant
intrinsic factors
Factors not directly related ot the purpose of the study that may impact study results
extraneous variables
Factors that exist in the environment or situation
extrinsic factors
Extraneous factors are often the source of
bias
confounding
Different research designs possess characteristics that will control for or introduce extraneous factors into the study
true
In general, experimental designs provide the best opportunity to demonstrate a ________ relationship between the independent and dependent variables.
cause and effect
Experimental study designs include which of the following?
n-of-1 designs
Quasi-experimental designs
Randomized controlled trials
factors associated with the environment or situation
extrinsic factors
factors associated with personal characteristics of the participant (
intrinsic factors
True experimental design =
randomized controlled trial (RCT)
the gold standard design
randomized controlled trial (RCT)
what is the point of the gold standard
protects against threats to internal validity and controls extraneous variables
hallmarkk features of RCT
Control group - comparison
Random assignment
What is a control group
comparison group that doesn’t receive intervention of interest (active intervention)
Single most effective way accounting for threats to internal validity (maturation & hx effects)
control group
Types of RCT designs
Parallel arm design
Crossover design
Randomized block trial
Randomized block, mixed design
most common, participants are randomized into different levels of the IV
parallel arm design
parallel arm design
most common, participants are randomized into different levels of the IV
between group design
parallel arm design
participants in one group are compared to those in another group
between-group design
assign participants to a “block” first and then randomly assign into treatment groups
randomized block trial
what are randomized block trial
assign participants to a “block” first and then randomly assign into treatment groups
participants are randomly assigned into a sequence and each sequence includes both treatments but in a different order
cross over design
what is a crossover design
participants are randomly assigned into a sequence and each sequence includes both treatments but in a different order
what is a washout period
time between treatment protocols to allow the effects of the first treatment to diminish before starting the second treatment
combo of random block & crossover; participants are blocked upfront before being randomized into a sequence of treatments
randomized block mixed design
what is randomized block mixed design
combo of random block & crossover; participants are blocked upfront before being randomized into a sequence of treatments
when should you use randomized block mixed design
Used when potential of creating an imbalance between groups is based on a prognostic factor like sex or there is an potential of order effect based on sequencing of treatments
includes both between and within subject comparisons
mixed design
parallel arm
participants are randomly assigned to one of two or more groups, with each group receiving a different treatment (or a placebo/control)
helps to directly compare the effects of different interventions
block design
ensures that each treatment group has an equal number of participants by assigning them in “blocks” or small groups, which helps maintain balance, particularly in smaller trials
cross over
each participant receiving multiple treatments in a sequential manner, with a washout period in between to eliminate carry-over effects of the previous treatment
resource-efficient since each participant serves as their own control, but it’s not suitable for treatments with long-lasting effects or conditions that could change significantly over time of their own accord
mixed design
incorporate elements from both crossover and parallel-arm designs, some participants undergoing a crossover design while others are in parallel groups
hybrid approach can offer a balance between the efficiency of crossover designs and the straightforward comparisons possible in parallel-arm designs
The MOST effective way to control for extraneous variables within a research study is to include:
control group
Which of the following are benefits is associated with random assignment? (check all that apply)
Will balance potentially prognostic indicators across groups
Allocates participants into study groups in a unbiased manner
Will create unequal groups
Helps to optimize internal validity
Helps to optimize internal validity
Will balance potentially prognostic indicators across groups
Allocates participants into study groups in a unbiased manner
In intervention studies, it is best practice to use which of the following for your control group?
Placebo
Standard of care
No treatment
Rest
standard of care
The RCT design most associated with a washout period is called a
crossover design
what are quasi-experimental studies
those that are missing a control group and/or random assignments
examples of quasi studies
Non-equivalent pretest-postest design
Single-group designs
Non-equivalent pretest-postest design
multiple group design including a control/comparison group
Does NOT have random assignment like in RCT
Used due to practical and or ethical coniderations that offers valuable information
Posess limitations & biases
one group designs
no control group and includes only one group
No random assignment but it measures at multiple time points
Valuable in assessing change over time controlling for temporal effects & provide richer context for interpreting intervention’s impact
what is the IV in single-group designs
time - pre and post
what are threats to IV in one group designs
Maturation effects
hx effects
how do you control for maturation effects in one group designs
through adding measurements at multiple timepoints before and/or after intervention
3 types of one group designs
one group pretest-posttest design
One-way repeated measure design
Time series designs
One group pre-test post-test design
follow 1 group of PTs over 2 time points
pre-intervention & post-intervention with one group receiving the same treatment
One group pre-test post-test design
focuses more on how effects manifest over time & includes measurements at multiple time points AFTER intervention of the DV
One-Way Repeated measures design
includes measurements at multiple time points BEFORE and AFTER intervention
Time series design
A quasi-experimental design is missing which of the following features? (check all that apply)
An independent variable
Random assignment
A control group
A target population
control & random assignment
Which of the following quasi-experimental study designs generally provides the highest level of internal validity?
One group pretest-posttest design
Two group pretest-posttest design
One-way repeated measures design
Time series
Two group pretest-posttest design
consists of one measurement before and after intervention
one group pre post design
Consists of multiple measurements before and after intervention
time series
Consists of one measurement before and multiple measurements after
One way repeated measures design
A quasi-experimental design is missing which of the following features? (check all that apply)
An independent variable
Random assignment
A control group
A target population
Random assignment
A control group
Which of the following quasi-experimental study designs generally provides the highest level of internal validity?
One group pretest-posttest design
Two group pretest-posttest design
One-way repeated measures design
Time series
Two group pretest-posttest design
Consists of one measurement before and after intervention
One group pretest postest design
Consists of multiple measurements before and after intervention
Time series
Consists of one measurement before and multiple measurements after
One way repeated measures design
Group design limitations
structure/requirements might not be practical or possible
Limited measurement points
Group means/averages can mask individual differences
single-subject design
study of 1 PT under controlled conditions
benefits of single subhect design
More feasible, weak external validity (cannot generalize from one subject), high internal validity,
what is iV & DV in single subect design
IV = time, DV = target behavior
a phase
no intervention/baseline
helps to understand natrual change of DV over time
what has to happen before b phase starts
Establish stability of variable (then implement B)
b phase
intervetnion given
this increases in single subject as phases are added
Internal validity
Ways to increase internal validity in signle subject designs
Phases to be equal (measurements)
Extending # of measures in each phase until stability is established
Adding second A and B phases to see changes to DV when intervention is taken away and implemented
When developing a single-subject design, the phases: (check all that apply)
Should be relatively equal in terms of length and the number of measurement points
Can consist of as little as two measurement points
Should be extended out until the stability of the variable is established
Should be extended out until the stability of the variable is established
Should be relatively equal in terms of length and the number of measurement points
Which of the following is true about single-subject designs?
External validity will always be low
External validity will always be high
Internal validity will always be low
Internal validity will always be high
External validity will always be low
Rank order the following experimental designs by their internal validity, starting with the study that generally offers the highest degree of internal validity.
AB single design
time series
crossover
one group pretest posttest design
1 - crossover design
2 - time series design
3 - one group-pretest post-test design
4 - AB Single subject design
based on RCT what design features would you expect to be included in this investigation?
A control (or comparison) group
Random assignment of participants into study groups
RCT design is primarily associated with a high degree of:
internal validity
As a reader/reviewer of a journal article, you would expect the exclusion criteria to identify the major factors or characteristics that would:
Likely confound the study results
Distinguish between the study groups
Describe the patient population
Likely confound the study results
As a reader/reviewer of a journal article, how would you determine if the authors have operationalized the independent variable sufficiently?
There are enough details to support that the groups are similar as the start of the study.
There are enough details for a clinician or researcher to replicate the treatment procedures.
There are enough details for the reader to understand there is a clear difference between the levels of the independent variable.
There are enough details to identify the tools used to measure the study outcomes.
There are enough details for a clinician or researcher to replicate the treatment procedures.
There are enough details for the reader to understand there is a clear difference between the levels of the independent variable.
The inclusion of a control group primarily improves _________ by controlling for __________.
External validity; history and maturation effects.
External validity; confounding variables.
Internal validity; history and maturation effects.
Statistical validity; confounding variables.
Internal validity; history and maturation effects.
Single-Subject A-B Design
inherent limitation of the study design used with this design
It will always suffer from low external validity
If the researchers wanted to enhance the internal validity of their investigation in signle subject designs which of the following could they do?
Increase the number of measurements within each phase.
Add more phases in the study.
Ensure they include males and females in the study.
Include more dependent variables.
Increase the number of measurements within each phase.
Add more phases in the study.
