Critical Evaluation Of Research Final Flashcards
What is Evidence-Based Practice, EBP, (3)?
- External Research Evidence
- Internal Clinical Experience
- Fully Informed Patient Preference
What is the scientific method (5)?
Step 1: Observe behavior
Step 2: Form a Hypothesis:
a tentative explanation of the observation
Step 3: Rational Method:
generate a testable prediction (based on the hypothesis)
Step 4: Data Collection:
evaluate the prediction with systematic, planned observations
Step 5: Support or Nullify:
use the observations to support, refute, or refine the original hypothesis
Describe what it means to say that science is empirical, public, and objective.
3 important PRINCIPLES of the Scientific Method:
- Empirical: answers are obtained by making observations.
(science requires empirical verification to be accepted) - Public:
• AVAILABLE: observations are available for evaluation by others.
- REPLICATION: repetition of observation allows verification of the findings.
- PEER REVIEW: evaluated by the researcher’s peers in scientific journals
- EVALUATE: erroneous concl. can occur either by error, chance, or fraud
- Objective:
“The dispassionate search for knowledge”
(the observations are structured so that the researcher’s biases and beliefs do not influence the outcome of the study)
Hypothesis
A TENTATIVE EXPLANANTION that is intended to be TESTED and CRITICALLY evaluated.
Must be:
• Testable
• Refutable, and
• Positive Statements (existence of change/new),
bc you cant conclude something doesn’t exist bc it wasn’t found
4 Foreground Elements
P (patient/problem)
I (intervention)
C (comparison/control)
O (outcome/effects)
In a research paper, where do you find:
How many individuals participated in the study, and what are their characteristics?
Method: Participants
In a research paper, where do you find:
Why was the study done?
Introduction: Literature Review
In a research paper, where do you find:
Did the study produce a statistically significant result?
Results: Data
In a research paper, where do you find:
The limitations of the study and directions for future research?
Discussion: Limitations
Humans are called _____________
Participants
Nonhumans are called ____________
Subjects
Do studies PROVE results?
NO! evidence only SUPPORTS findings (for or against)
Science is ___________ and ___________
incremental & cumulative
What is the Nuremberg Code?
A set of 10 guidelines for the ETHICAL treatment of human participants in research
What are 4 major ethical concerns/issues with research?
- No Harm:
• Must outline risk/benefit
• Benefit must be greater - Obtain Informed Consent:
• Purpose of study (blinding?)
• Right to decline participation @ any time
• Risk/Benefits
• Provide contact info for concerns - Use of Deception & Debriefing
• Passive Deception (withholding/omitting)
• Active Deception (misinformation/telling
lies), uses Confederates (in on the study)
• Debriefing, after study concludes, explanation/
purpose of the study to participants to
counteract/minimize harmful effects by deception - Confidentiality & Anonymity
• Confidentiality (info obtained is kept secret
and private)
• Anonymity (name is not linked/used w/ info
obtained @ any point)
Name several places where you can search for peer-reviewed articles
- ASHA
- Cochrane Library
- Campbell Collaboration
- MEDLINE
- NYU Library
- PLOS
- PsycINFO
- Scopus
Define Construct
• An intangible broad CONCEPT/IDEA/TOPIC
that helps EXPLAIN or PREDICT behavior in a theory
e.g. self-esteem, motivation, hunger, beauty, intelligence,
personality, learning, gravity, pain, anxiety (abstract)
• Can be influenced by external stimuli (variables) and, in
turn can influence external behaviors (dependent).
e.g.,
Stimulus –> Construct –> Behavior
(reward) –> (motivation) –> (Performance)
Define Variable
- Factors/Characteristics/Conditions that influence a construct
- A measurable representation of a construct
Give examples of Abstract Variables
intangible, not directly observable
- thoughts
- feelings
- perceptions
Give examples of Well Defined Variables
(easily observed and measured)
- age
- sex
- race
- height
- weight
Define Operational Definition
• Defines constructs in terms of HOW they will be
empirically measured
e.g., temperature (construct) will be measured by
Fahrenheit (O.D.)
______________ is demonstrated when scores obtained from a new measure are directly related to scores obtained from an established measure of the same variable.
Concurrent Validity
______________ is demonstrated when scores obtained from a measure accurately predict behavior according to a theory.
Predictive Validity
______________ requires that the scores obtained from a measurement procedure behave exactly the same as the variable itself.
This type of validity is based on many research studies that use the same measurement procedure and grows gradually as each new study contributes more evidence.
Construct Validity
Define Validity of Measurement
The degree to which a measurement procedure is measuring what it claims to measure
Define Reliability of Measurement
The stability or consistency of a measurement
Describe the relationship between Validity and Reliability as they pertain to measures
Validity requires reliability (not valid w/o it) BUT
Reliability does not require validity
WHY?
Bc the concept of reliability is based on the assumption that the variable measured is stable/constant.
Why are Operational Definitions sometimes necessary.
When you need to convert an ABSTRACT variable into a CONCRETE entity that can be observed, studied, and replicated
How may you operationally define
RECEPTIVE VOCAB.
• PPVT-5 (Peabody Picture Voc. Test)
• ROWPVT (Receptive One-Word Picture Vocabulary
Test)
- Y/N Questions
- Story Retell
How may you operationally define
EXPRESSIVE VOCAB.
- EVT-2 (Expressive Vocab. Test)
- EOWPVT-4 (Expressive One-Word Picture Vocab. Test)
- Picture Naming
- MLU
- Narratives
How may you operationally define
HOARSENESS
- altered vocal quality
- laryngeal tenseness
- loudness
- pitch
- vocal effort
- Perception of audible aperiodicity in sound (raspy)
How may you operationally define
BREATHINESS
- altered vocal quality
- laryngeal tenseness
- loudness
- pitch
- vocal effort
- Perception of audible air escape
Define Test-retest reliability
The CONSISTENCY of SCORES when administered on MULTIPLE OCCASIONS (same test multiple times)
Define Inter-rater reliability
The consistency of scores when DIFFERENT OBSERVERS measure (or score) the measurement (degree of agreement between raters)
A SLP has developed a new test for measuring receptive language and would like to determine the validity of the test. The new test and an established measure of receptive language are both administered to a sample of participants
Describe the pattern of results that would establish concurrent validity for a new test.
A POSITIVE correlation pattern between the new test and the well-established test would establish concurrent validity (and an UPWARD direction in a scatterplot).
What are the 4 SCALES of measurement and give examples?
- Nominal:
CATEGORIES that represent QUALITATIVE differences
• Gender (1 = F, 2 = M)
• Subject group (1 = CWS, 2 = CWNS) - Ordinal:
CATEGORIES that have DIFF NAMES, but organized
SEQUENTIALLY (Directional relationship)
• Academic Grades A-F (Ranked Data)
• Surveys 1 -5 from agree to disagree (Likert Scales) - Interval:
CATEGORIES organized SEQUENTIALLY and all
Categories/intervals are the SAME SIZE
• Temperature
• SAT scores - Ratio
CATEGORIES organized SEQUENTIALLY and all
Categories/intervals are the SAME SIZE
• Height
• Weight
Interval Scale
• Has an ARBITRARY zero point (e.g., temperature, year)
Ratio Scale
• Has a TRUE zero (e.g, % of verbs uttered or # of SS)
What is a ceiling effect
The clustering of scores at the HIGH end of a measurement scale
How can a ceiling effect be a problem?
It hinders the ability to measure changes with little or no possibility of increases in value.
What are demand characteristics?
POTENTIAL CUES or features of a study that:
- SUGGEST to the participants what the PURPOSE/HYPOTHESIS IS
- INFLUENCE the participants to RESPOND/BEHAVE in a certain way
How do Demand Characteristics limit the Validity of the measurements obtained in a research study?
It triggers participant REACTIVITY, which modifies their natural behavior in response to the study and thereby INFLUENCES the MEASURES they PRODUCE
What is a population?
The large group of interest to a researcher
What is a sample?
• The small set of individuals who participate
in the study
• Must be representative of the population
Describe the relationship between a population and a sample.
Circular:
• The Population (all the individual of interest)
• The Sample (selected from the population)
• The Study is conducted on the sample
• The Results are then generalized to the pop
Under what circumstances can a sample be biased?
When the sample differs in characteristics from those of the population (due to SELECTION BIAS)
List 5 Research Designs
- Descriptive
- Correlational
- Experimental
- Quasiexperimetnal
- Nonexperimental
Describe the concept of a study’s EXTERNAL validity.
The extent to which the results obtained in a research study hold true outside that specific study (generalizability).
Describe the concept of a study’s INTERNAL validity.
A research study has internal validity if it produces a single, unambiguous explanation for the relationship between two variables.
What are the common threats to INTERNAL validity?
A threat to internal validity is any factor that allows for an alternative explanation.
- Environmental variables:
• Time of day
• Room temperature,
• Amount of light… - Individual differences:
• Characteristics that differ from one person
to another - Time-related variables:
• History (events before/after study)
• Instrumentation (faulty)
• Regression to the Mean (the more x you
take a test the closer the score to the mean)
What are the common threats to EXTERNAL validity?
Any characteristic of a study that LIMITS the ability to GENERALIZE the results from a research study.
- Participants:
• Selection bias
• Volunteer bias
• Participant characteristics - Features of the Study:
• Novelty effect (emotions of participants)
• Multiple treatment interference:
- Practice Effect (test learning)
- Fatigue effects (timing of test)
• Experimenter characteristics
3.Measurements:
• Sensitization (demand characteristics)
• Generalizability across measures ( can it be
reduplicated)
• Time of measurement
Experimental research studies tend to have very (a) ________ internal validity but often have relatively (b) ________ external validity.
(a) High
(b) Low
Independent Variables
The variable that is MANIPULATED by the researcher
Dependent Variables.
The variable that is OBSERVED for changes to assess the effects of the Ind. Var.
Extraneous variables.
All OTHER variables
Define characteristics of a BETWEEN-SUBJECT design
- Nonequivalent group design
- Each GROUP is assigned to 1 TREATMENT condition
- Each individual in the study has ONLY 1 SCORE
• Compares scores from DIFFERENT GROUPS of
individuals
Define characteristics of a WITHIN-SUBJECT design
• Pre-post design
• The SAME GROUP of Individuals participate in ALL of
the TREATMENT conditions, but not necessarily in the
same order.
- Each individual in the study has MORE THAN 1 SCORE
- Compares scores from the SAME GROUP of individuals
HOW do INDIVIDUAL DIFFERENCES between groups threaten the internal validity of between-subject design?
• Individual differences can produce HIGH VARIABILITY
OF SCORE, making it difficult to see if the treatment
had an effect
• Each score is obtained from individuals who have
DIFF. UNIQUE CHARACTERISTICS
e.g., sex, age range, personality, Social Economic
Status
HOW does DIFFERENTIAL ATTRITION between groups threaten the internal validity of between-subject design?
• CHANGES THE MAKE UP the participants and in turn makes it difficult to see if effect change was due to the treatment conditions by differential attrition.
HOW does COMMUNICATION BETWEEN GROUPS threaten the internal validity of between-subject design?
• Diffusion:
Where the TRUE effects of the treatment are
masked by the shared information
> reduce the difference between the
two conditions.
• Compensatory Equalization
When the untreated group learns about
the treatment being received by the other group
and demands the same or equal treatment
> If the demand is met, the research study no longer
has a no-treatment condition for comparison.
3 primary techniques for limiting Confounding Individual Differences in B/S?
- Random Assignment
- Matching
- HOLD a variable CONSTANT
Within Subject VS. Between-Subjects
• Individual differences:
B/S higher between tx conditions
W/S reduced between tx conditions
• Time-related factors and order effects:
B/S not a problem for this type of study
W/S problem for this type of study
• Number of participants:
B/S more needed
W/S fewer needed
WITHIN SUBJECT DESIGN (PROS v CONS)
Pros:
• Fewer participants needed
• Reduces problems based on Indv.
differences (baseline & tx)
Cons:
• Susceptible to time-related factors
- Fatigue
- Practice Effects
- Envionmental changes btwn conditions
Describe HOW factors threaten the internal validity of a within-subject design?
- Confoundting from ENVIRONMENTAL Var.:
Where systemic changes occur due to change of rooms, change in room temperature, experimenter change,
which provides an alternative explanation for the differences between treatments
- Confounding from TIME-RELATED Var.:
• History, co-occurring events in their life may
contribute to diff between treatment conditions.
• Maturation:
Systemic change in physiology or psychology
• Instrumentation:
Decay or faulty
- Testing effects:
* Practice effects (due to learned/experience)
* Fatigue effects (exhaustion)
* Carryover effects (lingering effects from other Tx)
• Statistical regression:
where extreme scores on one measure move to a less
extreme in another
ALL provide an alternative explanation of scores other than the TX
3 measures of central tendency?
- Mean (average)
- Median (middle)
- Mode (most often)
2 measures of Variability
- Variance (avg. squared)
2. Standard Deviation (SD)
Mean
The average SCORE in the distribution
Standard Deviation (SD)
The average DISTANCE of scores from the mean
Correlation Coefficient
A NUMERICAL value that describes the Direction, Form, and Strength of the Relationship between variables
In a ___________ relationship, there is a tendency for two variables to change in the same direction; as one variable increases, the other tends to increase.
POSITIVE
In a___________ relationship, there is a tendency for two variables to change in opposite directions; an increase in one variable tends to be accompanied by a decrease in the other.
NEGATIVE
3 Broad Categories of Research Design (based on Q and Data structure)
- Examine individual variables
• Descriptive - Examine relationships between variables
by measuring two (or more) variables for each
participant
• Correlational - Examine relationships between variables
by comparing two (or more) groups/sets of scores.
• Experimental
• Quasi-experimental
• Non-experimental
Purpose of Research Designs (5)
- Descriptive Purpose:
Produce a description of INDIVIDUAL VARIABLES as they exist within a specific group. - Correlational Purpose:
Produce a description of the RELATIONSHIP BETWEEN two variables but does NOT attempt to EXPLAIN the relationship. (linear, curvilinear, positive, negative) - Experimental Purpose: Produce a CAUSE-AND -AFFECT explanation for the relationship between two variables.
- Quasiexperimental Purpose: Attempts to produce a cause-and-effect explanation, BUT FALLS SHORT
- Nonexperimental Purpose: Produce a description of the RELATIONSHIP BETWEEN two variables but does NOT attempt to explain the relationship.
4 Basic ELEMENTS of an Experimental Design
- Manipulation
- Measurement
- Comparison
- Control
Nonexperimental: How are groups/conditions defined
Naturally Occurring:
• Typically defined by a characteristic
• In terms of time (pre/post)
Quasiexperimental: How are groups/conditions defined
Naturally Occurring:
• Typically defined by a characteristic
• In terms of time (pre/post)
Experimental: How are groups/conditions defined
Manipulated/Created by the researcher
_________________ Developmental design evaluates changes in behavior related to age; by examining one group of participants who are roughly the same age, at different TIMES.
LONGITUDINAL
Researchers indicate that people who suffer from depression also tend to experience insomnia. However, it is unclear whether the depression causes insomnia or the lack of sleep causes depression. What problem is demonstrated by this example?
The directionality problem
How do Extraneous Variables become Confounding Variables?
When they’re transitions from random changes to systemic changes
3 Methods/Techniques for CONTROLLING extraneous variables.
- Holding a Variable Constant
- Matching Across Tx Conditions
- Control by Randomization
Holding a variable constant is a technique for removing one threat to (a) ________________, but it can limit the (b) _________________ of an experiment.
(a) Internal Validity
(b) External Validity
The Zimbardo prison study is an extreme example of a Stimulation Experiment.
Researchers often use Simulation Experiments in an attempt to increase the ____________ validity of the results.
EXTERNAL
Nonexperimental (less control)
How does this affect the internal and external validity?
Lower Internal Validity
Higher External Validity
Correlational (less control)
How does this affect the internal and external validity?
Lower Internal Validity
Higher External Validity
Field studies tend to have (a) __________ internal validity and (b)________ external validity than experimental research studies
(a) lower
(b) higher
Factorial Design
A research study involving two or more factors
• often referred to by THE NUMBER of its
factors, e.g., two-factor design or three-
factor design.
A 2 X 3 factorial design is a two-factor design with (a) _________ levels of the first factor and (b)_________
levels of the second factor.
(a) two
(b) three
Define Main Effect
- Each variable is looked at INDEPENDENTLY
- The Mean Differences among the Levels of 1 Factor
e.g., Diff. in mean BNT score between Tx A and Tx B
Define Interaction Between Factors
- Variables are looked at IN RELATION to each other
- Occur when two factors acting together produce mean differences that are NOT explained by the main effect of the two factors
- Occur when the effect of one factor depends on the different levels (or value) of a second factor
e.g., Diff. in mean scores on BNT for Tx A differ from Tx B depending on whether a person received treatment 1x/week or 5x/week
2 Statistics Categories
- Descriptive
Purpose: To ORGANIZE and SUMMARIZE
data to EXPLAIN and COMMUNICATE results to others - Inferential
Purpose: To INFER CHARACTERISTICS of a
POPULATION from a SAMLL SAMPLE of participants
Null Hypothesis (H )
0A statement about the population indicating that there is NO change, NO Difference, NO Effect, or NO Relationship
Alternative Hypothesis (H )
aThe Research Hypothesis that states there IS a Change, Difference, Effect, or Relationship
p < .05
Alpha Level = Statistical Significance Level
means there is a LESS THAN 5% chance that our DATA resulted from CHANCE
Effect Size (E.S.)
A STANDARDIZED measure of the MAGNITUDE of an observed EFFECT (size of the effect)
Cohen’s d (r =)
• r = .1 (small effect)—explains 1% of total variance
• r = .3 (medium effect)—explains 9% of total variance
• r = .5 (large effect)—explains 25% of total variance
ELEMENTS needed for Single-Case to qualify as an Experiment
MUST HAVE:
• Manipulation (of Independent Variables)
• Controls (strict control of Extraneous Variables)
• Cause-and-effect relationships between variables.
How are Single-Case Study Results Evaluated?
Visually by a simple graph of the data.
We want to see:
• Large DIFFERENCE between treatment conditions
• Small VARIABILITY in performance w/in ea treatment
condition
• STRICT CONTROL of all other variables
What name is given to a single-case design consisting of the following four phases in the order given:
baseline, treatment, baseline, treatment?
ABAB “Reversal” Design
Ethical concern for the ABAB design?
Withdraw of successful treatments
Describe the Structure of a Multiple-Baseline Design
• Begins with 2 Simultaneous Baseline Phases.
• Treatment Phase is introduced for the 1st baseline and
is delayed for 2nd baseline, where observations
continue
• Later, the treatment phase is initiated for the 2nd
baseline.
ID the Primary Strength of a Multiple-Baseline Design.
• It ELIMINATES the need for a reversal or return-to-
baseline, phase, and,
• Therefore, is WELL SUITED for evaluating treatment
EFFECTS that are permanent or long-lasting.
STATISTICAL TEST FOR:
A between-subjects study comparing TWO groups’ means
Independent-samples t-test
STATISTICAL TEST FOR:
A between-subjects study comparing means of MORE THAN TWO groups
ANOVA
STATISTICAL TEST FOR:
A within-subjects study comparing TWO groups’ means
Paired-samples t-test
Systematic Reviews vs. Meta-analyses vs. Traditional Literature reviews?
Systematic Reviews:
• A STANDARD PROCEDURE for Lit. Review
• OPERATIONALIZED criteria (including/excluding studies)
• MULTIPLE REVIEWERS independently evaluate sources
Meta-analyses:
• A type of Systemic Review w/ 1+ QUANTITATIVE STATS
• Effect Size
• Sample Size (weighted)
• Confidence Interval
Traditional Literature reviews:
• Decisions are made BY the author
• about WHICH studies to REVIEW and
• WHAT findings are RELEVANT to the study
A primary difference between Systematic Reviews and Meta-Analyses.
The primary difference is that Meta-Analysis provides a STATISTICAL (qualitative) ANALYSIS of systemically selected and evaluated data
How much WEIGHT should a patient’s preference carry in clinical decision making in the context of EBP?
In the context of evidence-based practice, fully informed patients’ preference should carry SIGNIFICANT WEIGHT. Ethical principles establish a patient’s AUTONOMY and BENEFICENCE as primary and of utmost importance. While the clinician’s responsibility is to educate the patient of the external research and potential cost-benefits of treatment, ultimately, the patient should feel empowered to make informed decisions about their treatment.
Operationalization reduces (a) __________ and increases the (b) ____________ of a study
(a) subjectivity
(a) reliability
______________ design evaluates changes in behavior related to age by examining different groups of individuals with each group representing a different AGE.
Cross-sectional
Nonexperimental and quasi-experimental studies always contain a threat to internal validity.
TRUE
Factors
independent variables
