Item Analysis and Test Reliability

Question 1

Three factors that affect the reliability coefficient

Answer 1

1. Content homogeneity
2. Range of scores
3. Guessing (T/F tests tend to have lower validity, due to easier guessing)

Question 2

What are the four main methods of assessing a test’s reliability?

Answer 2

1. Test-retest
2. Alternate Forms
3. Internal Consistency
4. Inter-rater

Question 3

_____ is due to random factors that affect the test performance of examinees in unpredictable ways and include distractions during testing, ambiguously worded test items, and examinee fatigue.

Answer 3

Measurement error

Question 4

_____ is the result of actual differences among examinees with regard to whatever the test is measuring. It’s assumed to be consistent, which means that an examinee’s true score will be the same regardless of which form of the test he or she takes or who scores the test.

Answer 4

True score variability

Question 5

_____ provides information about the consistency of scores over time.

Answer 5

Test-retest reliability

Question 6

_____ provides information about the consistency of scores over different forms of the test and, when the second form is administered at a later time, the consistency of scores over time.

Answer 6

Alternate forms reliability

Question 7

provides information on the _____ consistency of scores over different test items and is useful for tests that are designed to measure a single content domain or aspect of behavior. It’s not useful for speed tests because it tends to overestimate their reliability.

Answer 7

Internal consistency reliability

Question 8

What are three methods of evaluating a test’s internal consistency reliability?

Answer 8

1. Coefficient alpha (Cronbach’s alpha)
2. Kuder-Richardson 20 (KR-20)
3. Split-half reliability

Question 9

This method of evaluating internal consistency reliability involves administering the test to a sample of examinees and calculating the average inter-item consistency.

Answer 9

Coefficient alpha (Cronbach’s Alpha)

Question 10

This method of evaluating internal consistency reliability is an alternative to coefficient alpha that can be used when test items are dichotomously scored (e.g., as correct or incorrect).

Answer 10

Kuder-Richardson 20 (KR-20)

Question 11

This method of evaluating internal consistency reliability involves administering the test to a sample of examinees, splitting the test in half (often in terms of even- and odd-numbered items), and correlating the scores on the two halves.

Answer 11

Split-half reliability

Question 12

Given split-half reliability coefficients tend to underestimate a test’s reliability, due to using two shorter forms, they are usually corrected with the _____, which is used to determine the effects of lengthening or shortening a test on its reliability coefficient.

Answer 12

Spearman-Brown prophecy formula

Question 13

_____ provides information on the consistency of scores over different raters and is important for tests that are subjectively scored.

Answer 13

Inter-rater reliability

Question 14

What are two methods of evaluating inter-rater reliability?

Answer 14

1. Cohen’s kappa coefficient -
   used to assess the consistency of ratings assigned by two raters when ratings represent a nominal scale
2. Kendall’s coefficient of concordance - assess the consistency of ratings assigned by three or more raters when ratings represent ranks

Question 15

This occurs when two or more raters communicate with each other while assigning ratings, which results in increased consistency (but often decreased accuracy) in ratings and an overestimate of inter-rater reliability.

Answer 15

Consensual observer drift

Question 16

Tests that are _____ with regard to content tend to have larger reliability coefficients than tests that are _____, and this is especially true for internal consistency reliability.

Answer 16

homogeneous; heterogeneous

Question 17

Reliability coefficients are _____ when test scores are unrestricted in terms of range. An unrestricted range occurs when the examinees included in the sample are heterogeneous with regard to the characteristic(s) measured by the test – i.e., when the sample includes examinees who have high, moderate, and low levels of the characteristic(s).

Answer 17

Larger

Question 18

For dichotomously scored items, an item’s _____ indicates the percentage of examinees who answered the item correctly.

Answer 18

difficulty level (p)

Obtained by dividing the number of examinees who answered the item correctly by the total number of examinees.

p = .30 to .70 (moderate difficulty) is preferred on most tests.

Question 19

With regard to _____, the optimal p value lies halfway between 1.0 and the probability that the item can be answered correctly by guessing.

Answer 19

guessing

Question 20

The _____ ranges from -1.0 to +1.0 and indicates the difference between the percentage of examinees with high total test scores (often the top 27%) who answered the item correctly and the percentage of examinees with low total test scores (the bottom 27%) who answered the item correctly.

Answer 20

discrimination index (D)

For most tests, a D value of .30 or higher is acceptable. Note that an item’s difficulty level affects its ability to discriminate, with items of moderate difficulty having higher levels of discrimination.

Question 21

When a test’s reliability coefficient is less than 1.0, this means that an examinee’s obtained test score may or may not be his or her true score. Consequently, obtained test scores are often interpreted in terms of a _____, which indicates the range within which an examinee’s true score is likely to be given his or her obtained score

Answer 21

confidence interval

Question 22

The \_\_\_\_\_t is used to construct a confidence interval, and it’s calculated by:
Standard Deviation (√[1-reliability coefficient])

Answer 22

standard error of measurement

For a 68% confidence interval, you add and subtract one standard error of measurement to and from the obtained score; for a 95% confidence interval, you add and subtract two standard errors of measurement; and for a 99% confidence interval, you add and subtract three standard errors of measurement.

Question 23

In Item Response Theory, _____ is indicated by the slope of the item characteristic curve (ICC)

Answer 23

Discrimination

Question 24

In Item Response Theory, _____ indicated by the percent of low-, average-, and high-ability examinees who answered the item correctly.

Answer 24

Difficulty level

Question 25

In Item Response Theory, _____ is indicated by the point at which the ICC crosses the y-axis: The closer this point is to 0, the more difficult it is for examinees to choose the correct answer to the item just by guessing.

Answer 25

Probability of guessing correctly