Module 3: Reliability and Validity Flashcards
What is Reliability?
Consistency/stability of measurement.
What is Validity?
The extent to which a test measures what it is meant to measure.
What is the Classical Test Theory?
The assumption that the score of each test taker reflects their actual ability, plus a degree of measurement error.
What is the Equation for the Concept of Error?
Observed score = true score + sources of error.
What is Random Error?
Source of error caused by unpredictable fluctuations and inconsistencies of other variables in the measurement process.
What is Systematic Error?
Source of error that is typically constant or proportionate to what is presumed to be the true value of the variable being measured.
What are the Four Sources of Error Variance?
- Test construction (sampling of content/poor item wording).
- Test administration (due to test-taker, test-environment, test-examiner).
- Test scoring and interpretation (objective vs subjective scoring).
- Other sources of error (sampling error, methodological error, social desirability bias/impression management).
What is the Standard Error of Measurement (SEM)?
Estimates the degree of error associated with the observed score.
Equation: SEM = SD x square root of 1.0 - r.
What are the Confidential Intervals and SEM?
68% = 1 SEM.
95% = 2 SEM.
99% = 3 SEM.
What is Reliability Coefficient?
Index of reliability, a proportion that indicates the ratio between the true score variance on a test and the total variance.
How is Variance Measured?
Standard deviation squared.
What are the Four Main Types of Reliability?
- Consistency across time (test-retest).
- Consistency across different versions (alternative/parallel forms).
- Consistency across individual items of a test (internal consistency).
- Consistency across scorers (inter-rater reliability).
What is Test-Retest Reliability?
If a construct being measured remains unchanged, then similar scores will be observed at time 1 and time 2.
The longer time passes, this reliability coefficient can lower.
Subject to practice/familiarity effects.
What are Alternative/Parallel Forms?
Degree to which different versions of a test/measure correlative.
Expensive and time-consuming.
Avoids practice effects.
What is Internal Consistency?
Degree to which different items in a test correlate with each other.
Not content, just consistency (how similar they are?).
What is Inter-Related/Inter-Scorer Reliability?
Degree to which two or more scorers correlate in measure.
Scoring criteria must be clear and judges should be well-trained.
Coefficient = cohen’s kappa (k).
At a Minimum, What Should all Tests Have?
Content validity.
Internal consistency reliability.
What is Ecological Validity?
Extent to which a study is realistic or representative of real life.
What is Psychometric Soundness?
Evaluation of both reliability and validity.
What is Face Validity?
Extent to which the scale looks like what it claims to measure.
What is Content Validity?
Extend to which items represent the construct universe.
What is Criterion-Related Validity?
How well the test correlated with external criteria or outcome (real world)?
Does it have predictive power?
What are the Two Types of Criterion-Related Validity?
Concurrent: compare test score with criterion measured at the same time.
Predictive: compare score with a criterion ar some point in the future.
What is Construct Validity?
Testing the correlation between the construct and other theoretically related constructs (nomological network).
What are the Two Ways of Demonstrating Construct Validity?
Convergent: tests should correlate strongly with existing measure of same construct.
Divergent: tests should not correlate with a construct that is not theoretically related.
What is Test Bias?
Systematically prevents accurate and impartial measurement.
What are the Types of Measurement Bias?
Intercept bias: consistently over/under-predicting.
Slope bias: shows weaker correlation for specific groups compared to others.
Rating error: (un)intentional judgement error resulting from misuse of scale.
What is Test Fairness?
Extent to which a test is used in an impartial. just and equitable way.
What is a Nomological Net?
Describes what other constructs your construct is related to.
