Chapter 4: Reliability

Question 1

Reliability

Answer 1

Consistency or stability of test scores

Question 2

Factors that impact reliability

Answer 2

When the test is administered
Items selected to be included
External distractions (ex- noise)
Internal distractions (ex- fatigue)
Person administering test
Person scoring test

Question 3

Two components of score

Answer 3

True score (representative of true knowledge or ability)
Error score

Question 4

Systematic error

Answer 4

Error resulting from receiving a different set of instructions for test

Question 5

Classical test theory equation

Answer 5

Xi=T+E
Xi- obtained score
T- true score
E- error

Question 6

What measurement error reduces

Answer 6

Usefulness of measurement
Generalizability of test results
Confidence in test results

Question 7

Content sampling error

Answer 7

Difference between sample of items on test and total domain of items

Question 8

How good sampling affects error

Answer 8

Reduces it

Question 9

Largest source of measurement error

Answer 9

Content sampling error

Question 10

Time sampling error

Answer 10

Random fluctuations in performance over time

Can be due to examinee (fatigue, illness, anxiety, maturation) or due to environment (distractions, temperature)

Question 11

Inter-rater differences

Answer 11

When scoring is subjective, different scorers may score answers differently

Question 12

Clerical errors

Answer 12

Adding up points incorrectly

Question 13

Reliability (mathematic definition)

Answer 13

Symbol: rxx

Ratio of true score variance to total score variance (number from 0 to 1, where 0 is total error and 1 is no error)

Question 14

Reliability equation

Answer 14

rxx= (sigma^2T)/(sigma^2X)

Question 15

Reliability’s relation to error

Answer 15

Greater the reliability, the less the error

Question 16

What reliability coefficients mean

Answer 16

rxx of 0.9: 90% of score variance is due to true score variance

Question 17

Test-retest reliability

Answer 17

Administer the same test on 2 occasions
Correlate the scores from both administrations
Sensitive to sampling error

Question 18

Things to consider surrounding test-retest reliability

Answer 18

Length of interval between testing
Activities during interval (distraction or not)
Carry-over effects from one test to next

Question 19

Alternate-form reliability

Answer 19

Develop two parallel forms of test
Administer both forms (simultaneously or delayed)
Correlate the scores of the different forms
Sensitive to content sampling error (simultaneous and delayed) and time sampling error (delayed only)

Question 20

Things to consider surrounding alternate-form reliability

Answer 20

Few tests have alternate forms

Reduction of carry-over effects

Question 21

Split-half reliability

Answer 21

Administer the test
Divide it into 2 equivalent halves
Correlate the scores for the half tests
Sensitive to content sampling error

Question 22

Things to consider surrounding split-half reliability

Answer 22

Only 1 administration (no time sampling error)
How to split test up
Short tests have worse reliability

Question 23

Kuder-Richardson and coefficient (Cronbach’s) alpha

Answer 23

Administer test
Compare each item to all other items
Use KR-20 for dichotomous answers and Cronbach’s alpha for any type of variable
Sensitive to content sampling error and item heterogeneity
Measures internal consistency

Question 24

Inter-rater reliability

Answer 24

Administer test
2 individuals score test
Calculate agreement between scores
Sensitive to differences between raters

Question 25

Composite scores

Answer 25

Scores that are combined to form a combined score

Reliability of these is usually better than their individual parts

Question 26

Difference scores

Answer 26

Calculated difference between 2 scores
Reliability of these is usually lower than their individual parts (information is lost: only can see change, not initial baseline)

Question 27

Choosing a reliability test to use

Answer 27

Multiple administrations: test-retest reliability

One administration: homogeneous content uses coefficient alpha and heterogeneous content uses split-half coefficient

Question 28

Factors to consider when evaluating reliability coefficients

Answer 28

Construct being measured
Time available for testing
How the scores will be used
Method of estimating reliability

Question 29

High-stake decision tests: reliability coefficient used

Answer 29

Greater than 0.9 or 0.95

Question 30

General clinical use: reliability coefficient used

Answer 30

Greater than 0.8

Question 31

Class tests and screening tests: reliability coefficient used

Answer 31

Greater than 0.7

Question 32

How to improve reliability

Answer 32

Increase number of test items
Use composite scores
Develop better items
Standardize administration

Question 33

Standard error of measurement (SEM)

Answer 33

Standard deviation of test administered to the same individual an infinite number of times
Useful when interpreting test scores
When reliability increases, this decreases

Question 34

How to calculate confidence intervals

Answer 34

Use SEM and SD

Question 35

Relationship between reliability and confidence interval

Answer 35

Reliability increases, confidence interval decreases

Question 36

Test manuals/researchers report: information included

Answer 36

Internal consistency
Test-retest
Standard error of measurement (SEM)
Information on confidence intervals

Question 37

Generalizability theory

Answer 37

Shows how much variance is associated with different sources of error