Chapter 10- Rigor and Validity in Quantitative Research Flashcards
Validity
“the appropriate truth of an inference”
inferences that a cause results in a hypothesized effect are valid to the extent that researchers can marshal strong supporting evidence
Validity is always a matter of degree, not an absolute
Validity is the property of an inference- not of a research design, but design elements profoundly affect the inferences that can be made.
Threats to validity
reasons that an inference could be wrong
when researchers introduce design features to minimize potential threats, the validity of the inference about relationships under study is strengthened.
statistical conclusion validity
concerns the validity of inferences that there truly is an empirical relationship or correlation, between the presumed cause and effect.
The researcher’s job is to provide strong evidence that an observed relationship is real.
internal validity
concerns the validity of inferences, that, given that an empirical relationship exists, it is the independent variable, rather than something else, that caused the outcome.
Reserachers must develop strategies to rule out the plausability that some factor other than the independent variable accounts for the observed relationship
construct validity
involves the validity of inferences “from the observed persons, settings, and cause and effects operations included in the study to the constructs that this instance might represent:
one aspect concerns the degree to which an intervention is a good representation of the underlying construct that was theorized as having the potential to cause beneficial outcomes.
another issue concerns whether the measures of the outcomes are good operationalizations of the construct for which it is intended.
external validity
concerns whether inferences about observed relationships will hold over variations in persons, setting, or time.
relates to the generalizability of the inferences- a critical concern for evidence-based nursing practice
Controlling Confounding Participant characteristics
1) Randomization
2) Crossover- participants serve as their own controls
3) Homogeneity-participants are homogenous with respect to confounding variables. results can not be generalized to tpye of people who did not participate in the study
4) Stratification/Blocking
5) Matching
6) Statisitical control
Statistical Conclusion Validity
statistical methods are used to support inferences about whether relationships exist
Researchers can make design decisions that protect against reading false statistical conclusions
Low statistical Power
statistical power-
When small sample sizes are used, statistical power tends to be low and the analyses may fail o show that the IV and DV are related- even when they are.
maximizing precision- which is achieved through accurate measuring tools, controls over confounding variables, and powerful statistical methods.
Internal Validity
The extent to which it is possible to make an inference that the IV, rather than another factor, truly had the causal effect on the outcome (DV)
If researchers do not manage confounding variation, the conclusions that the outcome was caused by the independent variable is open to challenge
6 threats to internal validity- each threat represent an alternative explanation that competes with the independent variable as the cause of the outcome.
Temporal ambiguity
the cause most precede the effect
establishing temporal sequencing may be difficult in correlational studies- it may be unclear whether the IV preceded the DV, or vice versa.
This is especially true in cross-sectional studies
Selection threat
emcompasses biases resulting from preexisting differences between groups.
When not selected at randon, the groups copared are seldom completed equivalent
Selection bias is the most problematic and frequent threat to internal valididty in studies not using an experimental design
History threat
Concerns the occurrence of external events that take place concurrently with the IV and that can affect outcomes.
Maturation threat
processes occurring during the study as a result of the passage of time rather than as a results of the IV
a one-group pretest-posttest design is highly suspective to this threat
refers to any change that occurs as a function of time
Mortality/Attrition threat
Individuals dropping out of a stop
attrition bias can occur in single-group quasi-experiments if those dropping out of the study are biased subset that makes it look like a change in average values resulting from the treatment
prospective cohort study- may be differential attributions between groups being compared because of death, illness, relocation.
the longer the study, the greater the risk
Testing and Instrumental threat
Testing- the effects of takign a pretest on the posttest performance.
sensitization more likely to occur when exposed to controversial or novel material in the pretest
instrumentation- bias reflects changes in measuring instruments or methods of measurements between two points of data collect.
Can occur even if the same measurement tool is used (data collector more experienced, participants bored)
Construct validity
the first step in fostering construct validity is a careful explication of the treatment, outcomes, setting and population constructs of interest.
the next step is to select instances that match those constructs as closely as possible.
further cultivated when researchers assess the match between the exemplars and the constructs and the degree to which any “slippage” occurred
has most often been a concern to researchers in connection with the measurements of outcomes
it is just as important for the IV to be a strong instance of the construct of interest as it is for the measured outcome to have a strong correspondence to the outcome construct.
in nonexperimental research, researchers do not create and manipulate the hypothesized cause, so ensuring construct validity of the IV is often difficult
covers outcomes, treatments, persons and settings
there are constructs that require careful description and the selection of good exemplars that match those constructs
Threats to construct validity
Reasons that inferences from a particular study exemplar to an abstract constuct could be erroneous.
Such a threat could occur if the operationalization of the construct fails to incorporate all teh relevant characteristics of the underying construct or if it includes extraneous content
Reactivity to the study situation threat
Participants may behave in a particular manner because they are aware of their role in a study Hawthorne effect)
these perceptions become an unwanted part of the treatment construct under study.
Researcher expectancies
A similar threat stems from researcher’s influence on participant responses through subtle communication about desired outcomes.
Novelty effects
when a treatment is new, participants and research agents alike might alter their behavior.
compensatory effects
in intervention studies, compensatory equalization can occur if health care staff or family members try to compensate for the control group members’ failure to receive a perceived beneficial treatment
treatment diffusion or contamination
alternative treatment conditions can become blurred, which can impede good construct descriptions of the independent variable.
this may occur when participants in a control group condition receive services similar to those in the treatment condition.
or when those in the treatment group drop out, putting them in the control group.
external validity
concerns the extent to which it can be inferred that relationships observed in a study hold true over variations in people, conditions, and settings.
emerged as a major concern in the EBP world in which there is an interest in generalizing evidence from tightly controlled research settings to real-world clinic practice settings
may ask: whether a relationship observed in a study can be generalized to a larger population.
REPRESENTATIVENESS of the participants used in the study
REPLICATION- multi-site studies are powerful because more confidence in the generalizability of the results can be attained if findings are replicated in several sites.
systematic reviews are a crucial aid to external validity precisely because they illuminate the consistency of results in studies replicated with different groups and settings.
threats to external validity
threats to external validity concern ways in which relationships between variables might interact with or be moderated by variations in people, settings, time, and conditions.
Interactions between relationships and people- external validity threat
an effect observed with certain types of people might not be observed with other types of people.
A common complaint about RCTs is that many people are excluded- not because they would not benefit from the treatment, but because they cannot provide needed research data (do not speak Engligh, cognitive impairment) or they would not allow the best test for the intervention (complex comorbidities)
Quiz info
Content validity concerns the degree to which an instrument has an appropriate sample of items for the construct being measured.
Criterion-related validity examines the relationship between scores on an instrument and an external criterion.
Construct validity examines these questions: What is this instrument really measuring?
Does it validly measure the abstract concept of interest? Face validity refers to whether an instrument looks as though it is measuring the appropriate construct; while this is good, it is not the most important measure of validity.
Reliability and validity are interrelated but not the same.
External validity threats
Rationale:Common threats to external validity are reactivity (how a subject reacts to being studied); experimenter (when a characteristic of the researcher influences the study result); and novelty (when the knowledge of what is being done is new somehow effects the outcome, either favorably or unfavorably).
Maturation and attritions are threats to internal validity.
