Chapter 5

Question 1

Refers to consistency in measurement.

Not an all-or-none matter.

A test may be reliable
in one context and unreliable in another.

Answer 1

RELIABILITY

Question 2

Index of reliability, a proportion
that indicates the ratio between the true score variance on a
test and the total variance.

Answer 2

Reliability coefficient

Question 3

Score on an ability test is presumed to reflect
not only the testtaker’s true score on the ability being measured but also error.

X= T + E

Answer 3

Classical test theory

Question 4

A statistic useful in describing sources of test score variability

Answer 4

Variance

Question 5

2 types of measurement error

Answer 5

Systematic
Random

Question 6

Source of error in measuring a targeted variable
caused by unpredictable fluctuations and inconsistencies of other
variables in the measurement process.

Answer 6

Random error

Question 7

Source of error in measuring a variable that
is typically constant or proportionate to what is presumed to be the true
value of the variable being measured.

Doesn’t affect score consistency

Answer 7

Systematic error

Question 8

Source of variance which refers to variation among items
within a test as well as to variation among items between tests.

Differences are sure to be found in the way the items are worded
and in the exact content sampled.

Answer 8

Item sampling
or Content sampling

Question 9

The testtaker’s reactions to those influences are the source of
one kind of error variance:

Sources of error in test administration

Answer 9

Test environment - physical setting where a test is administered

Testtaker variables - factors related to the individual taking the test

Examiner-related variables - factors related to the person administering the test

Question 10

2 possible sources of error variance under test scoring and interpretation

Answer 10

Scorers - subjectivity involved
Scoring systems - glitch if computer scoring

Question 11

3 potential sources of nonsystematic error in assessment
situation

Answer 11

Forgetting
Failing to notice abusive
behavior
Misunderstanding instructions regarding
reporting

Question 12

Estimate of reliability obtained by
correlating pairs of scores from the same people on two different
administrations of the same test.

Answer 12

Test-retest reliability

Question 13

The
estimate of test-retest reliability when the interval between testing is greater than six months

Answer 13

Coefficient
of stability

Question 14

The degree of the relationship between various forms of a test can be
evaluated by means of _______________; often referred to as coefficient of equivalence.

Answer 14

Alternate-forms or parallel-forms
coefficient of reliability

Question 15

Exist when, for each form of the test, the
means and the variances of observed test scores are equal.

Answer 15

Parallel forms

Question 16

Simply different versions of a test that have
been constructed so as to be parallel.

Answer 16

Alternate forms

Question 17

Obtained by correlating two pairs of scores obtained from
equivalent halves of a single test administered once.

Answer 17

Internal consistency estimate of reliability

or

Estimate of interitem consistency

Question 18

Step 1. Divide the test into equivalent halves.

Step 2. Calculate a Pearson r between scores on the two halves
of the test.

Step 3. Adjust the half-test reliability using the Spearman-Brown formula

Answer 18

3 steps in the computation of coefficient of split-half reliability

Question 19

3 acceptable ways to split a test

Answer 19

Randomly assign
Odd-even reliability
By content

Question 20

Assign odd-numbered
items to one half of the test and even-numbered items to the
other half

Answer 20

Odd-even reliability

Question 21

Each half contains items equivalent with respect to content and
difficulty

Answer 21

By content

Question 22

Randomly assign items
to one or the other half of the test.

Answer 22

Randomly assign

Question 23

Formula that allows a test developer or user to
estimate internal consistency reliability from a correlation of two halves of
a test.

Answer 23

Spearman-Brown

Question 24

Degree of correlation among all
the items on a scale.

Answer 24

Inter-item consistency

Question 25

Degree to which a test measures a single factor.

Answer 25

Homogeneity

Question 26

Degree to which a test measures different factors.

Answer 26

Heterogeneity

Question 27

Statistic of choice for determining the inter-item consistency of dichotomous items, primarily those items that can be scored right or wrong

Answer 27

KuderRichardson formula 20 or KR-20

Question 28

May be thought of as the mean of all possible split-half correlations, corrected by the Spearman-Brown formula. Typically ranges in value from 0 to 1.

Answer 28

COEFFICIENT ALPHA

Question 29

A measure used to evaluate the internal consistency of a test that focuses on the degree of difference that exists between item scores.

Answer 29

AVERAGE PROPORTIONAL DISTANCE (APD)

Question 30

Step 1: Calculate the absolute difference between scores for all of the items. Step 2: Average the difference between scores. Step 3: Obtain the APD by dividing the average difference between scores by the number of response options on the test, minus one

Answer 30

Steps in getting AVERAGE PROPORTIONAL DISTANCE (APD)

Question 31

Degree of agreement or consistency between two or more scorers (or judges or raters) with regard to a particular measure.

Answer 31

Inter-scorer reliability

Question 32

1. the test items are homogeneous or heterogeneous in nature 2. the characteristic, ability, or trait being measured is presumed to be dynamic or static 3. the range of test scores is or is not restricted 4. the test is a speed or a power test 5. the test is or is not criterion-referenced

Answer 32

5 natures of test

Question 33

Homogeneity vs heterogeneity of a test

Answer 33

Homogeneity of test items - all the items on a test are very similar in content, difficulty, and format, essentially measuring the same construct Heterogeneity of test items - items vary significantly in content, difficulty, and format, potentially assessing multiple related but distinct aspects of a concept

Question 34

Trait, state, or ability presumed to be ever-changing as a function of situational and cognitive experiences.

Answer 34

Dynamic characteristic

Question 35

Trait, state, or ability presumed to be relatively unchanging such as intelligence.

Answer 35

Static characteristic

Question 36

When a time limit is long enough to allow testtakers to attempt all items, and if some items are so difficult that no testtaker is able to obtain a perfect score

Answer 36

Power test

Question 37

Generally contains items of uniform level of difficulty (typically uniformly low) so that, when given generous time limits, all testtakers should be able to complete all the test items correctly

Answer 37

Speed test

Question 38

A reliability estimate of a speed test should be based on performance from two independent testing periods using one of the following: ___________ from two separately timed half tests

Answer 38

Test-retest reliability Alternate-forms reliability Split-half reliability

Question 39

Designed to provide an indication of where a testtaker stands with respect to some variable or criterion, such as an educational or a vocational objective.

Answer 39

Criterion-referenced test

Question 40

Any theoretical and statistical framework describing how respondents generate their answers to items on a scale or instrument and explaining associated sources of error

Answer 40

PSYCHOMETRIC MODELS

Question 41

3 major approaches of psychometric models:

Answer 41

Classical test theory Generalizability theory Item response theory

Question 42

A framework of principles and assumptions about how to determine the reliability of a set of data.

Answer 42

GENERALIZABILITY THEORY

Question 43

Include things like the number of items in the test, the amount of training the test scorers have had, and the purpose of the test administration.

Answer 43

Facets

Question 44

Examines how generalizable scores from a particular test are if the test is administered in different situations

Answer 44

Generalizability study

Question 45

Represents the influence of particular facets on the test score, which is similar to reliability coefficient in the true score model.

Answer 45

Coefficients of generalizability

Question 46

Involves the application of information from the generalizability study Developers examine the usefulness of test scores in helping the test user make decisions.

Answer 46

Decision study

Question 47

States that the probability that an item will be answered correctly is a function of an underlying trait or ability that is not directly observable; that is, a latent trait; also referred to as latent-trait theory

Answer 47

ITEM RESPONSE THEORY

Question 48

Test items or questions that can be answered with only one of two alternative responses, such as true– false, yes–no, or correct–incorrect questions

Answer 48

Dichotomous test items

Question 49

Test items or questions with three or more alternative responses, where only one is scored correct or scored as being consistent with a targeted trait or other construct

Answer 49

Polytomous test items

Question 50

Provides a measure of the precision of an observed test score

Answer 50

Standard error of measurement

Question 51

Determine whether a score is significantly different from a criterion

Answer 51

Confidence interval

Question 52

Statistical measure that can aid a test user in determining how large a difference should be before it is considered statistically significant.

Answer 52

Standard error of the difference

Question 53

4 sources of error variance

Answer 53

Test construction Test administration Test scoring and interpretation Other sources of error

Question 54

2 other sources of error variance

Answer 54

Sampling error - not an actual representative of a population of data Methodological error - mistake or flaw in the design or execution of a research study