Reliability Flashcards
What is reliability in measurement?
Consistency in measurement.
What does a reliability coefficient represent?
It indicates the ratio between true score variance and total variance.
What does a reliability coefficient of 0 indicate?
No reliability.
What does a reliability coefficient of 1 indicate?
Perfect reliability.
What is the formula for observed score?
Observed score = true score + error (X = T + E).
What does Cronbach’s Alpha indicate if it’s too high?
Redundancy in test items.
What is measurement error?
Factors affecting the measurement process, aside from the variable being measured.
What is a true score?
A score without measurement error, though it can’t be directly observed.
How can an approximation of a true score be obtained?
By averaging multiple measurements.
What are two influences that interfere with measuring a true score?
Time between measurements and the act of measurement itself.
What is a carry-over effect?
When the measurement process alters what is being measured.
How can practice effects in testing be minimized?
By using different sets of tests measuring the same construct.
What is the fatigue effect in testing?
Reduced motivation or mental energy due to repeated testing.
How is a true score related to a construct score?
A true score is tied to the measurement tool, while a construct score is tied to a theoretical variable.
What is random error?
Unpredictable fluctuations and inconsistencies in the measurement process.
What is systematic error?
Consistent or proportionate error related to the true value being measured.
How is reliability related to variance in test scores?
Higher true score variance relative to total variance indicates higher reliability.
What does a reliability of 0.80 imply?
80% of score variance is due to true differences.
What is total observed variance in terms of reliability?
True variance + error variance.
What does item sampling refer to in test construction?
Variations within items to measure a specific construct, such as physical aggression.
What factors in test administration can cause measurement error?
Testing environment, test-taker conditions, and examiner behavior.
What is test-retest reliability?
A reliability estimate based on the correlation between scores on two administrations of the same test.
What is parallel form reliability?
Two identical versions of a test designed to measure the same construct with equal statistical properties.
What is internal consistency reliability?
The consistency of items within a test, often measured with split-half or Cronbach’s Alpha.
What does the Spearman-Brown formula do?
Adjusts reliability for split-half testing.
What does a reliability coefficient of 0.7 signify?
An acceptable level of reliability according to Kline (2015).
What is the purpose of a reliability coefficient?
To provide an index of error from test construction, administration, or scoring.
What is the Classical Test Theory formula for observed score?
X = E + T (error + true score).
What is Domain Sampling Theory?
A method estimating reliability based on sampling of test items and inter-item consistency.
What does Generalizability Theory aim to describe?
Specific sources of variation in test scores due to testing conditions.
What does Item Response Theory (IRT) analyze?
Item difficulty and item discrimination in assessing construct measures.
What is Cronbach’s Coefficient Alpha used for?
It is used to measure internal consistency for tests with interval or ratio data, especially in Likert-scale formats.
What range do values of Cronbach’s Alpha typically fall within?
Values range from 0 to 1, with higher values indicating greater reliability.
What does a Cronbach’s Alpha of 0.99 suggest?
Potential redundancy in test items.
What does a reliability coefficient of 0.80 mean in terms of measurement error?
20% of score variability is due to measurement error.
What is the coefficient of inter-rater reliability used for?
To provide information about scoring error between multiple raters.
What does the coefficient of stability assess?
Error due to time intervals between test administrations.
What does the coefficient of equivalence assess?
Error from the instrument (test items) itself.
What is a true score according to Classical Test Theory (CTT)?
The score that reflects an individual’s actual ability or trait level as measured by the test.
What does Domain Sampling Theory suggest about test items?
Reliability increases as the number of items sampled increases.
What is the goal of Generalizability Theory?
To identify specific sources of variation in test scores within the testing environment.
What does “facets” refer to in Generalizability Theory?
Variables such as test item number, scorer training, and test purpose that define a test’s “universe.”
What does Item Response Theory (IRT) focus on in terms of test items?
The difficulty level and discrimination ability of test items.
What is meant by “level of difficulty” in IRT?
The degree to which an item is challenging or easy for test-takers.
What does “level of discrimination” refer to in IRT?
An item’s ability to differentiate between individuals with high or low levels of the measured trait.
How is reliability in a test related to true score and error variance?
Higher true score variance relative to total variance indicates higher reliability.
What does a test with low reliability suggest about observed scores?
They are mostly determined by measurement error rather than true scores.
How does the Classical Test Theory (CTT) view measurement error?
CTT assumes measurement error is always present in any test score.
What happens to test-retest reliability over long intervals?
It tends to decrease, especially over intervals longer than six months.
What is the coefficient of stability in test-retest reliability?
The reliability estimate for intervals over six months.
How does parallel form reliability differ from alternate form reliability?
Parallel forms have identical statistical properties; alternate forms are similar but may vary slightly in mean and variance.
What is inter-item consistency?
The degree of relatedness among items within a test.
What does the Spearman-Brown formula help estimate?
Internal consistency reliability for split-half tests.
When is the Kuder-Richardson Formula 20 used?
For determining inter-item consistency in dichotomous (binary) items.
What is the main aim of Item Response Theory (IRT)?
To analyze item properties and assess the construct’s ability being measured through item difficulty and discrimination.
