Week 4 Flashcards
Validity
the degree to which a test or instrument measures what it purports
Validity
Logical / Face Validity
do the methods and approach make sense?
Validity
Content Validity
Does the test fully represent the domain of the concept it is intended to measure?
Validity
Criterion Validity - Concurrent
correlating an instrument with a criterion administered at the same time
Validity
Criterion Validity - Predictive
correlating an instrument with a criterion administered in the future
Criterion Validity - Predictive
Cross- Validation
- generate a predictive equation with 1/2 of the sample
- confirm the prediction equation works with other 1/2 of sample
Validity
Construct Validity
degree to which the scores from a test measure the hypothesis construct
Measurement Reliability
- refers to the consistency and stability of a measurement tool or method
- indicates how dependably a method measures something over time, across different observers, or in various contexts
What are common sources of measurement errors?
- testing
- instrumentation
- scoring
- participant
Reliability Tests
Test-retest Reliability
measures the consistency of results when the same test is administered to the same sample at different points in time
Reliability Tests
Interrater Reliability
assesses the degree to which different observers or rates agree in their assessments
Reliability Tests
Parallel Forms
comparing two different versions of a test that are designed to be equivalent
Reliability Tests
Internal Consistency
examine the consistency of results across items within a test
* a common measure of internal consistency is Cronbach’s alpha, which assesses how well the items on a test measure the same construct
Common Statistical Approaches
Test-retest Reliability
- Pearson correlation coefficient
- Intraclass correlation coefficient (ICC)
- Coefficient of Variation
Common Statistical Approaches
Interrater Reliability
- Percent agreement
- Cohen’s alpha
- Intraclass correlation coefficient (ICC)
Common Statistical Approaches
Parallel Forms Reliability
- Pearson Correlation coefficient
- Split-Half method
- Counterbalancing method
Common Statistical Approaches
Internal Consistency
- Cronbach’s Alpha
- Split-Half reliability
Ranking research design from weakest evidence to strongest evidence.
- Animal research (does not involve animals)
- Clinical textbooks, experts opinions (not primary research)
- Case reports / case studies (no or little design)
- Case-control studies (observational)
- Cohort studies (observational)
- Randomized control trials (experimental)
- Systematic review (evidence study)
- Meta-analysis (evidence study)
Threats to Internal Validity
History
External or historical event that occurred during the course of the study that may be responsible for the effects instead of the intervention itself
Threats to Internal Validity
Maturation
An internal and natural process that leads participants to change on the dependent measure
Internal Threats to Validity
Testing
Changes in test scores occur because of repeated testing
Threats to Internal Validity
Differential Selection
Any difference between the study groups before the start of the study due to either self-selection or experimenter selection
Threats to Internal Validity
Instrumentation
Changes in instrument calibration or human observers
Threats to Internal Validity
Regression to the Mean
The phenomenon that if a variable is extreme on its first measurement, it will tend to be closer to the average on its second measurement
Threats to Internal Validity
Selection Interactions / Expectancy
Both researchers or study subjects may tend to expect one group may perform better (or worse) the other
Threats to Internal Validity
Experimental Mortality (Attrition)
People can drop out of the study for any reason; if reason is related to treatment status it may cause bias
What are threats to External Validity?
- selection bias
- pretesting
- experimental setting
- multiple treatment
Threats to External Validity
Selection Bias
- Who is included/excluded
- Race/ethnicity, age, sex, lifestyles, SES
- Disease characteristics
Threats to External Validity
Pretesting
If you pretest, then when participants take the “true test” their responses may differ compared to if they never took the a pretest
Threats to External Validity
Experimental Setting
- Participants performance improves just b/c they know they are being watched
- Hawthorne Effect
Threats to External Validity
Multiple Treatments
Receiving on treatment may influence how you respond to other treatments
Random Control Trial (RCT)
- A study design where participants are randomly assigned to either the intervention group or the control group
- randomization minimizes bias, allowing for causal inferences
Quasi-Experimental
- A study design that looks at the effect of an intervention without random assignment
- while it can still compare groups, the lack of randomization makes it more prone to bias
Natural Experiment
- A study that takes advantage of “naturally occurring circumstances in which subsets of the population have different levels of exposure to a supposed causal factor, in a situation resembling an actual experiment”
- Provides an alternative when it is impossible or unethical to assign treatment
