Chapter 5: Reliability

Question 1

_____ is a synonym for dependability or consistency.

Answer 1

Reliability

Question 2

A _____ is an index of reliability, a proportion that indicates the ratio between the true score variance on a test and the total variance.

Answer 2

reliability coefficient

Question 3

Recall from our discussion of _____ that a score on an ability test is presumed to reflect not only the testtaker’s true score on the ability being measured but also error.

Answer 3

classical test theory

Question 4

A statistic useful in describing sources of test score variability is the _____(σ2)—the standard deviation squared.

Answer 4

variance

Question 5

Variance from true differences is true variance, and variance from irrelevant, random sources is _____.

Answer 5

error variance

Question 6

The term _____ refers to the proportion of the total variance attributed to true variance. The greater the proportion of the total variance attributed to true variance, the more _____ the test.

Answer 6

reliability/reliable

Question 7

1) Test construction
2) Test administration
3) Test scoring and interpretation
4) Other sources of error: Underreport and Overreport

Answer 7

Sources of Error Variance (4)

Question 8

Sources of Error Variance:

One source of variance during test construction is item sampling or content sampling, terms that refer to variation among items within a test as well as to variation among items between tests.

Answer 8

Test construction

Question 9

Sources of Error Variance:

test environment: the room temperature, the level of lighting, and the amount of ventilation and noise, for instance.

testtaker variables. Pressing emotional problems, physical discomfort, lack of sleep, and the effects of drugs or medication can all be sources of error variance.

Examiner-related variables: physical appearance, demeanor, presence, absence, oral exam emphasizing key words, nonverbal cues when correctness

Answer 9

Test administration

Question 10

Sources of Error Variance:

The advent of computer scoring and a growing reliance on objective, computer-scorable items virtually have eliminated error variance caused by scorer differences in many tests. If subjectivity is involved in scoring, then the scorer (or rater) can be a source of error variance.

Answer 10

Test scoring and interpretation

Question 11

Reliability Estimates (4)

Answer 11

1) Test-Retest Reliability Estimates
2) Parallel-Forms and Alternate-Forms Reliability Estimates
3) Split-Half Reliability Estimates
4) Other Methods of Estimating Internal Consistency:
a) Inter-item consistency
b) The Kuder-Richardson formulas
c) Coefficient alpha

Question 12

Reliability Estimates:

using the same instrument to measure the same thing at two points in time.

is an estimate of reliability obtained by correlating pairs of scores from the same people on two different administrations of the same test.

**The passage of time can be a source of error variance. The longer the time that passes, the greater the likelihood that the reliability coefficient will be lower.

**even when the time period between the two administrations of the test is relatively small, various factors (such as experience, practice, memory, fatigue, and motivation) may intervene and confound an obtained measure of reliability

Answer 12

Test-Retest Reliability Estimates

Question 13

Reliability Estimates:

it is referred to as an internal consistency estimate of reliability or as an estimate of inter-item consistency.

Ex. Both groups take both tests: group A takes test A first, and group B takes test B first. The results of the two tests are compared, and the results are almost identical, indicating high parallel forms reliability.

Put simply, you’re trying to find out if test A measures the same thing as test B.

source of error variance: item sampling
cons: time-consuming and expensive.

Answer 13

Parallel-Forms and Alternate-Forms Reliability Estimates

Question 14

Reliability Estimates:

is obtained by correlating two pairs of scores obtained from
equivalent halves of a single test administered once.

One acceptable way to _____ is to randomly assign items to one or the other half of the test.

odd-even reliability

**The Spearman-Brown formula

Answer 14

Split-Half Reliability Estimates

Question 15

Reliability Estimates: Other Methods of Estimating Internal Consistency

refers to the degree of correlation among all the items on a scale. A measure of inter-item consistency is calculated from a single administration of a single form of a test. An index of interitem consistency, in turn, is useful in assessing the homogeneity of the test.

Tests are said to be homogeneous if they contain items that measure a single trait.

The more homogeneous a test is, the more _____ it can be expected to have.

Answer 15

Inter-item consistency

Question 16

Reliability Estimates: Other Methods of Estimating Internal Consistency

Dissatisfaction with existing split-half methods of estimating reliability compelled to develop their own measures for estimating reliability.

a measure of internal consistency reliability for measures with dichotomous choices

**coefficient alpha or coefficient α－20.

Answer 16

The Kuder-Richardson formulas (Kuder-Richardson formula 20, or KR-20)

Question 17

Reliability Estimates: Other Methods of Estimating Internal Consistency

In contrast to KR-20, which is appropriately used only on tests with dichotomous items, _____ is appropriate for use on tests containing “nondichotomous items”.

is the preferred statistic for obtaining an estimate of internal consistency reliability. Coefficient alpha is widely used as a measure of reliability, in part because it requires only one administration of the test.

Answer 17

Coefficient alpha

Question 18

Reliability Estimates: Other Methods of Estimating Internal Consistency

Unlike a Pearson r, which may range in value from －1 to ＋1, coefficient alpha typically ranges in value from _____.

Answer 18

0 to 1

Question 19

Variously referred to as scorer reliability, judge reliability, observer reliability, and inter-rater reliability, inter-scorer reliability is the degree of agreement or consistency between two or more scorers (or judges or raters) with regard to a particular measure.

problem is a lack of clarity in scoring criteria, solution: rewrite the scoring criteria section of the manual to include clearly written scoring rules, group discussion, and practice exercises and information on rater accuracy.

Answer 19

Measures of Inter-Scorer Reliability

Question 20

Perhaps the simplest way of determining the degree of consistency among scorers in the scoring of a test is to calculate a coefficient of correlation. This correlation coefficient is referred to as a _____.

Answer 20

coefficient of inter-scorer reliability

Question 21

Using and Interpreting a Coefficient of Reliability

Three approaches to the estimation
of reliability: (3)

Answer 21

1) test-retest
2) alternate or parallel forms, and
3) internal or inter-item consistency

Question 22

Another question that is linked in no trivial way to the purpose of the test is, “How high should the coefficient of reliability be?” Perhaps the best “short answer” to this question is: “On a continuum relative to the purpose and importance of the decisions to be made on the basis of _____ on the test”.

Answer 22

scores

Question 23

The Nature of the Test

considerations concerning the purpose and use of a reliability coefficient are those concerning the nature of the test itself: (5)

Answer 23

1) test items are homogeneous or heterogeneous in nature;
2) the characteristic, ability, or trait being measured is presumed to be dynamic or static;
3) the range of test scores is or is not restricted;
4) the test is a speed or a power test; and
5) the test is or is not criterion-referenced

Question 24

1. Test-retest, 2, 1, administration, Pearson r or Spearman rho
2. Alternate-forms, 1 or 2, 2, test construction or administration, Pearson r or Spearman rho
3. Internal Consistency, 1, 1, test construction, Pearson r
4. Inter-scorer, 1, 1, scoring and interpretation, Pearson r or Spearman rho

Answer 24

Type of reliability, # of test forms, source of Error Variance, Statistical procedure

Question 25

Homogeneity vs heterogeneity of test items

Answer 25

homogenous = high internal consistency
heterogenous = low internal consistency

Question 26

Dynamic vs static characteristics

Answer 26

Dynamic = test-retest
Static = test-retest or alternate forms

Question 27

Restriction or inflation of range

Answer 27

Restricted = low correlation coefficient
Inflation = high correlation coefficient

Question 28

Speed tests vs power tests

Answer 28

Power = long time limit, difficult items
Speed = fair difficulty or low, test-retest, alternate-forms, split-half timed test

Question 29

A _____ is designed to provide an indication of where a testtaker stands with respect to some variable or criterion, such as an educational or a vocational objective.

Answer 29

Criterion-referenced tests

Question 30

Unlike norm-referenced tests, criterion-referenced tests tend to contain material that has been mastered in a hierarchical fashion.

Scores on criterion-referenced tests tend to be interpreted in _____ (or, perhaps more accurately, “master-failed-to-master”) terms, and any scrutiny of performance on individual items tend to be for diagnostic and remedial purposes

Answer 30

pass–fail

Question 31

Alternatives to the True Score Model (2)

Answer 31

1) Generalizability theory

2) Item response theory

Question 32

Alternatives to the True Score Model:

According to _____, given the exact same conditions of all the facets in the universe, the exact same test score should be obtained. This test score is the universe score, and it is, as Cronbach noted, analogous to a true score in the true score model.

examines how generalizable scores from a particular test are if the test is administered in different situations.

Answer 32

generalizability theory (G theory)

Question 33

Alternatives to the True Score Model:

_____ procedures provide a way to model the probability that a person with X ability will be able to perform at a level of Y.

because the construct being measured may be a trait (it could also be something else, such as an ability), a synonym for _____ in the academic literature is latent-trait theory.

Examples of two characteristics of items within an IRT framework are the difficulty level of an item and the item’s level of discrimination;

Answer 33

Item response theory

Question 34

two characteristics of items within an IRT framework

Answer 34

“Difficulty” in this sense refers to the attribute of not being easily accomplished, solved, or comprehended.

In the context of IRT, discrimination signifies the degree to which an item differentiates among people with higher or lower levels of the trait, ability, or whatever it is that is being measured.

Question 35

Reliability and Individual Scores (2)

Answer 35

1) The Standard Error of Measurement (SEM)

2) The Standard Error of the Difference between Two Scores

Question 36

Reliability and Individual Scores:

provides a measure of the precision of an observed test score. Stated another way, it provides an estimate of the amount of error inherent in an observed score or measurement.

The higher the reliability of a test (or individual subtest within a test), the lower the _____.

is the tool used to estimate or infer the extent to which an observed score deviates from a true score. We may define the _____ as the standard deviation of a theoretically normal distribution of test scores obtained by one person on equivalent tests.

Also known as the standard error of a score and denoted by the symbol σmeas , the standard error of measurement is an index of the extent to which one individual’s scores vary over tests presumed to be parallel.

Answer 36

Standard Error of Measurement

Question 37

Reliability and Individual Scores:

Comparisons between scores are made using the standard error of the difference, a statistical measure that can aid a test user in determining how large a difference should be before it is considered statistically significant.

Answer 37

Standard Error of the Difference between Two Scores