Test construction (Reliability)

Question 1

From the perspective of ____ test theory, variability in test scores reflects two factors: true differences between examinees on the attribute measured by the test and differences due to _____

Answer 1

classical; measurement (random) error

Question 2

Reliability is a measure of the amount of variability in obtained test scores that is due to _____ variability

Answer 2

true score

Question 3

A test’s reliability is commonly estimated by calculating a reliability coefficient, which is a type of _____ coefficient

Answer 3

correlation

Question 4

The reliability coefficient ranges in the value from _______ and is interpreted directly as a measure of _______ variability

Answer 4

0 to +1.0; true score

Question 5

If a test has a reliability coefficient of .91, this means that ___% of variability in obtained test scores is due to _____ variability, while the remaining 9% reflects _______

Answer 5

91; true score; measurement error

Question 6

Test-retest reliability is assessed by administering a test to the same group of examinees at two different _______ and them _____ two sets of scores

Answer 6

times; correlating

Question 7

The test-retest reliability coefficient is also known as the coefficient of _______

Answer 7

stability

Question 8

An alternate forms reliability coefficient is calculated by administering two _____ of a test to the same group of examinees and correlating the two sets of scores

Answer 8

equivalent forms

Question 9

The alternate forms reliability coefficient is also referred to as the coefficient of ______

Answer 9

equivalence (or equivalence and stability when there is a long period of time between administration of the two forms)

Question 10

A ______ reliability coefficient is calculated by splitting the test in half and correlating examinees’ scores on the two halves

Answer 10

split half

Question 11

Because the size of a reliability coefficient is affected by test length, the split-half method tends to _____ a test’s true reliability

Answer 11

underestimate

Question 12

The ______ formula is often used in conjunction with split-half reliability to obtain an estimate of what a test’s true reliability is

Answer 12

Spearman-Brown

Question 13

Coefficient ______, another method used to assess internal consistency reliability, indicates the average inter-item consistency rather than the consistency between two halves of the test

Answer 13

alpha

Question 14

The Kuder-Richardson Formula 20 can be used as a substitute for coefficient alpha when test items are scored ______

Answer 14

dichotomously

Question 15

Split-half reliability, coefficient alpha, and KR-20 are not appropriate for speed tests because they tend to _____ the reliability of these tests

Answer 15

overestimate

Question 16

Inter-rater reliability should be assessed whenever a test is ______ scored

Answer 16

subjectively

Question 17

The scores assigned by different raters can be used to calculate a ______ coefficient– for example, the _______ statistic which can be used when ratings represent a nominal or ordinal scale of measurement.

Answer 17

correlation (reliability); kappa

Question 18

Alternatively, percent agreement between raters can be calculated. A problem with this approach is that the resulting index of reliability can be artificially inflated by the effects of ______

Answer 18

chance agreement

Question 19

The magnitude of a reliability coefficient is affect by several factors. In general, the longer a test, the ______ its reliability coefficient

Answer 19

larger

Question 20

The _____ formula is used to estimate the effects of lengthening or ______ a test on its reliability coefficient.

Answer 20

Spearman-Brown; shortening

Question 21

If the new items do not represent the same content domain as the original items or are more susceptible to measurement error, this formula is likely to _____ the effects of lengthening the test

Answer 21

overestimate

Question 22

Like other correlation coefficients, the reliability coefficient is affected by the range of scores: The greater the range, the _______ the reliability coefficient

Answer 22

larger

Question 23

To maximize a test’s reliability coefficient, the tryout sample should include people who are _____ with regard the attributes measured by the test

Answer 23

heterogeneous

Question 24

A reliability coefficient is also affects by the probability that an examinee can select the correct answer to a test question simply by guessing. The easier it is to guess the correct answer, the ______ the reliability coefficient

Answer 24

smaller

Question 25

While the reliability coefficient is useful for assessing the amount of variability in test scores that is due to ____ variability for a group of examinees, it does not directly indicate how much we can expect an individual examinee’s obtained score to reflect his or her true score. The standard error of _______ is useful for this purpose.

Answer 25

true score; measurement

Question 26

The standard error of measurement is calculated by multiplying the standard deviation of test scores by the ________ of one minus the reliability coefficient

Answer 26

square root

Question 27

If a test’s standard deviation is 10 and its reliability coefficient is .91, the standard error of measurement is equal to ______

Answer 27

3.0

Question 28

The standard error of measurement is used to construct a ____ interval around an examinee’s obtained (“measured”) score

Answer 28

confidence

Question 29

In terms of magnitude, the standard error of the difference between two scores is always _____ than the SEM of either score because it reflects measurement error from both test scores

Answer 29

larger