Test construction

Question 1

Classical Test Theory and Reliability is based on the assumption that

Answer 1

obtained test scores (X) are due to a combination of true score variability (T) and measurement error (E): i.e., X = T + E.

Question 2

measurement error is due to

Answer 2

random factors that affect the test performance of examinees in unpredictable ways

Question 3

Test reliability refers to

Answer 3

the extent to which a test provides consistent information

Question 4

interpretation of reliability coefficient

Answer 4

They’re always interpreted directly as the amount of variability in obtained test scores that’s due to true score variability. For instance, if a test has a reliability coefficient of .80, this means that 80% of variability in obtained test scores is due to true score variability and the remaining 20% is due to measurement error.

Question 5

4 Methods for Estimating Reliability:

Answer 5

There are four main methods for assessing a test’s reliability: test-retest, alternate forms, internal consistency, and inter-rater.

Question 6

Test-retest reliability provides information about

Answer 6

the consistency of scores over time.

Question 7

test-retest reliability involves

Answer 7

administering the test to a sample of examinees, re-administering the test to the same examinees at a later time, and correlating the two sets of scores.

Question 8

Test-retest reliability is useful for tests that are designed to measure a characteristic that’s ___over time.

Answer 8

stable

Question 9

Alternate forms reliability provides information about

Answer 9

the consistency of scores over different forms of the test and, when the second form is administered at a later time, the consistency of scores over time.

Question 10

alternate forms reliability involves

Answer 10

administering one form of the test to a sample of examinees, administering the other form to the same examinees, and correlating the two sets of scores. Alternate forms reliability is important whenever a test has more than one form.

Question 11

Internal consistency reliability provides information

Answer 11

on the consistency of scores over different test items and is useful for tests that are designed to measure a single content domain or aspect of behavior.

Question 12

best reliability measures for measuring speed/performance

Answer 12

internal consistency relaibility is NOT a good measure because it tends to overestimate their reliability. For speed tests, test-retest and alternate forms reliability are appropriate.

Question 13

3 key methods for INTERNAL CONSISTENCY reliability

Answer 13

Coefficient Alpha (Cronbach’s Alpha)

Kuder-Richardson 20 (KR-20)

Split-Half Reliability:

Question 14

coefficient/cronbach’s alpha used for tests with ___

Answer 14

multiple response formats

Question 15

kuder-richardson 20 (KR-20) is used for tests with ___

Answer 15

dichotomous (e.g. right/wrong) scoring

Question 16

issue with split-half reliability is

Answer 16

a split-half reliability coefficient underestimates a test’s reliability and is usually corrected with the Spearman-Brown prophecy formula, which is used to determine the effects of lengthening or shortening a test on its reliability coefficient.

Question 17

Inter-rater reliability is important for measures that are ____scored and provides information on the consistency of scores or ratings assigned by different raters.

Answer 17

subjectively

Question 18

methods used to evaluate inter-rater reliability:

Answer 18

Percent agreement

Cohen’s kappa coefficient is also known as the kappa statistic

Question 19

issue with percent agreement method

Answer 19

Percent agreement can be calculated for two or more raters. A problem with this method is that it does not take chance agreement into account, which can result in an overestimate of reliability.

Question 20

when ratings represent a nominal scale, best inter rater reliability method is

Answer 20

Cohen’s kappa coefficient is also known as the kappa statistic and is one of several inter-rater reliability coefficients that is corrected for chance agreement between raters.

Question 21

The reliability of subjective ratings can be affected by _____It occurs when two or more raters communicate with each other while assigning ratings, which results in increased consistency (but often decreased accuracy) in ratings and an overestimate of inter-rater reliability.

Answer 21

consensual observer drift.

Question 22

3 Factors that Affect the Reliability Coefficient

Answer 22

content homogeneity
range of scores
guessing

Question 23

when a test’s reliability coefficient is .81, the reliability index is the ____

Answer 23

square root of .81, which is .90.

Question 24

For dichotomously scored items, an item’s difficulty level (p) indicates the percentage of examinees who

Answer 24

answered the item correctly.

Question 25

when 50 of 100 examinees answered an item correctly, the item’s p value is

Answer 25

50/100, or .50.

Question 26

For mastery tests (tests used to identify examinees who have mastered a certain level of knowledge or skill), ___ p values are preferred.

Answer 26

lower

Question 27

The item discrimination index (D) ranges from ___and indicates the difference between the percentage of examinees with ___ and the percentage of examinees with __

Answer 27

-1.0 to +1.0 ; high total test scores (often the top 27%) who answered the item correctly; low total test scores (the bottom 27%) who answered the item correctly.

Question 28

what is the optimal difficulty level for a four-answer multiple choice question

Answer 28

the chance of choosing the correct answer to a four-answer multiple-choice question by guessing is .25, and the optimal difficulty level for this type of item is calculated by adding 1.0 to .25 and dividing the result by 2: (1.0 + .25)/2 = 1.25/2 = .625.

Question 29

when 90% of examinees in the high-scoring group and 20% of examinees in the low-scoring group answered an item correctly, the item’s D value is

Answer 29

.90 minus .20, which is .70.

Question 30

an item’s difficulty level affects its ability to discriminate, with items of ___ having higher levels of discrimination.

Answer 30

moderate difficulty

Question 31

if a test has a standard deviation of 5 and a reliability coefficient of .84, its standard error of measurement equals

Answer 31

5 times the square root of 1 minus .84: 1 minus .84 is .16, the square root of .16 is .4, 5 times .4 is 2. In other words, when a test’s standard deviation is 5 and its reliability coefficient is .84, its standard error of measurement is 2.