Unit 5: Reliability

Question 1

measurement processes that alter what is measured

Answer 1

Carryover effects

Question 2

Portion of variability in test scores that is due to factors unrelated to the construct being measured.

Answer 2

error variance

Question 3

allows us to estimate, with a specific level of confidence, the range in which the true score is likely to exist

Answer 3

Standard Error of Measurement (SEM)

Question 4

the degree to which a measure predictably overestimates or underestimates a quantity

refers to the degree to which systematic error influences the measurement

Answer 4

bias

Question 5

an estimate of the extent to which these different forms of the same test have been affected by item sampling error, or other error

Answer 5

alternate forms reliability

Question 6

refers to the portion of variability in tests scores that reflects the actual differences in the trait, ability, characteristics the test is designed to measure

Answer 6

true variance

Question 7

An estimate of test-retest reliability may be most appropriate in gauging the reliability of tests that employ outcome measures such as r___

Answer 7

reaction time or perceptual judgments

Question 8

If interested in looking in the truth independent of measurement, psychologists look for the

Answer 8

construct score

Question 9

a person’s standing on a theoretical variable independent of any particular measurement

Answer 9

construct score

Question 10

In ability tests, __ are carryover effects in which the test itself provides an opportunity to learn and practice the ability being measured

Answer 10

practice effects

Question 11

a statistic that quantifies reliability, ranging from 0 (not at all reliable) to 1 (perfectly reliable)

Answer 11

reliability coefficient

Question 12

It provides an estimate of the amount of error inherent in an observed score or measurement

Answer 12

Standard Error of Measurement (SEM)

Question 13

based on the idea that a person’s test scores vary from testing to testing because of variables in the testing situation

Answer 13

generalizability theory

Question 14

to evaluate the relationship between different forms of a measure

Answer 14

alternate forms

Question 15

an estimate of the reliability of a test can be obtained without developing an alternate form of the test and without having to administer the test twice to the same people

Answer 15

Internal consistency estimate of reliability or estimate of inter-item consistency

Question 16

provides a measure of the precision of an observed test score

Answer 16

Standard Error of Measurement (SEM)

Question 17

if the test is __ in items, an estimate of internal consistency might be low relative to a more appropriate estimate of test-retest reliability

Answer 17

heterogeneous

Question 18

terms that refer to variation among items within a test as well as to variation among items between tests

Answer 18

item sampling or content sampling

Question 19

If two scores each contain error such that in each
case the true score could be higher or lower, then we would want the two scores to be further apart before we conclude that there is a significant difference between them

Answer 19

standard error of difference

Question 20

designed to provide an indication of where a testtaker stands with respect to some variable or criterion, such as an educational or a vocational objective

Answer 20

criterion-referenced test

Question 21

The tool used to estimate or infer the extent to which an observed score deviates from a true score

Answer 21

Standard Error of Measurement (SEM)

Question 22

Scores on criterion-referenced tests tend to be interpreted in -

Answer 22

pass–fail (or, perhaps more accurately, “master–failed-to-master”) terms

Question 23

Universe is described in terms of its facets, which include considerations such as —

Answer 23

the number of items in the test,
the amount of training the test scorers have had, and
the purpose of the test administration

Question 24

In general, a primary objective in splitting a test in half for the purpose of obtaining a split-half reliability estimate is to create what might be called __

Answer 24

mini- parallel-forms

Question 25

nature of the test

Answer 25

(1) The test items are homogeneous or heterogeneous in nature; (2) The characteristic, ability, or trait being measured is presumed to be dynamic or static; (3) The range of test scores is or is not restricted; (4) The test is a speed or a power test; and (5) The test is or is not criterion-referenced

Question 26

Refers to the degree of correlation among all the items on a scale

Answer 26

inter-item consistency

Question 27

often used when coding nonverbal behavior

Answer 27

inter-scorer reliability

Question 28

It is a specific application of a more general formula to estimate the reliability of a test that is lengthened or shortened by any number of item

Answer 28

Spearman-Brown formula

Question 29

The extent to which a testtaker’s score is affected by the content sampled on a test and by the way the content is sampled (i.e., the way in which the item is constructed) is a source of error variance

Answer 29

test construction

Question 30

A reliability estimate of a __ test should be based on performance from two independent testing periods

Answer 30

speed

Question 31

purpose: to evaluate the stability of a measure

Answer 31

test-retest reliability

Question 32

In many tests, the advent of computer scoring and a growing reliance on objective, computer-scorable items have virtually eliminated error variance caused by scorer difference

Answer 32

test scoring and interpretation

Question 33

it accurately measures internal consistency under highly specific conditions that are rarely met in real measures

Answer 33

cronbach's alpha

Question 34

The relationship between the SEM and the reliability of a test is _

Answer 34

inverse the higher the reliability of a test (or individual subtest within a test), the lower the SEM

Question 35

its use is typically to evaluate the homogeneity of a measure (or, all items are tapping in a single construct)

Answer 35

internal consistency

Question 36

exist when, for each form of the test, the means and the variances of observed test scores are equal

Answer 36

parallel forms

Question 37

By determining the reliability of one half of a test, a test developer can use the __ to estimate the reliability of a whole test

Answer 37

Spearman-Brown formula

Question 38

Computation of a coefficient of split-half reliability:

Answer 38

Step 1: Divide the test into equivalent halves. Step 2: Calculate a Pearson r between scores on the two halves of the test Step 3: Adjust the half-test reliability using the Spearman–Brown formula

Question 39

Tests designed to measure one factor, such as one ability or one trait,

Answer 39

homogeneous

Question 40

If the variance of either variable in a correlational analysis is restricted by the sampling procedure used, then the resulting correlation coefficient tends to be __

Answer 40

lower

Question 41

Valid tests give scores that closely approximate __ scores

Answer 41

construct

Question 42

allows a test developer or user to estimate internal consistency reliability from a correlation between two halves of a test

Answer 42

Spearman-Brown formula

Question 43

two types of variance

Answer 43

True variance: variance from true differences Error variance: variance from irrelevant, random sources

Question 44

most frequently used measure of internal consistency, but has several well- known limitations

Answer 44

cronbach's alpha

Question 45

seek to estimate the extent to which specific sources of variation under defined conditions are contributing to the test score

Answer 45

domain sampling theory

Question 46

If test developers or users wish to shorten a test, the __ may be used to estimate the effect of the shortening on the test’s reliability

Answer 46

Spearman–Brown formula

Question 47

refers to the proportion of the total variance attributed to true variance

Answer 47

reliability

Question 48

an estimate of the extent to which item sampling and other errors have affected test scores on versions of the same test when, for each form of the test, the means and variances of observed test scores are equal

Answer 48

parallel forms reliability

Question 49

test items or questions that can be answered with only one of two alternative responses, such as true–false, yes– no, or correct–incorrect questions

Answer 49

dichotomous test items

Question 50

a reference to an IRT model with specific assumptions about the underlying distribution

Answer 50

rasch model

Question 51

can be used to set the confidence interval for a particular score or to determine whether a score is significantly different from a criterion (such as the cutoff score of 70 described previously)

Answer 51

Standard Error of Measurement (SEM)

Question 52

__ are carryover effects in which repeated testing reduces overall mental energy or motivation to perform on a test

Answer 52

Fatigue effects

Question 53

consists of unpredictable fluctuations and inconsistencies of other variables in the measurement process; sometimes referred to as “noise”;

Answer 53

random error

Question 54

the goal of psychological assessment is to

Answer 54

maximize true variance minimize error variance

Question 55

statistical procedure of test-retest

Answer 55

pearson r or spearman rho

Question 56

trait, state, or ability presumed to be ever-changing as a function of situational and cognitive experiences

Answer 56

dynamic characteristic

Question 57

The influence of particular facets on the test score is represented by _

Answer 57

coefficients of generalizability

Question 58

approach to reliability evaluation

Answer 58

test-retest method

Question 59

trait, state, or ability presumed to be relatively unchanging, such as intelligence

Answer 59

static characteristic

Question 60

The total variance in an observed distribution of test scores (σ2)

Answer 60

equals the sum of the TRUE variance ( 𝜎2) and the ERROR variance ( 𝜎2)

Question 61

Given the exact same conditions of all the facets in the universe, the exact same test score should be obtained ▪ This test score is the __, and it is, as Cronbach noted, analogous to a true score in the true score model

Answer 61

universe score

Question 62

an estimate of reliability obtained by correlating pairs of scores from the same people on two different administrations of the same test

Answer 62

test-retest reliability

Question 63

A __ of behavior, or the universe of items that could conceivably measure that behavior, can be thought of as a hypothetical construct

Answer 63

domain

Question 64

obtaining estimates of alternate-forms reliability and parallel-forms reliability is similar in two ways to obtaining an estimate of test-retest reliability:

Answer 64

(1) Two test administrations with the same group are required (2) Test scores may be affected by factors such as motivation, fatigue, or intervening events such as practice, learning, or therapy

Question 65

the degree of the relationship between various forms of a test

Answer 65

Coefficient of equivalence

Question 66

generally contains items of uniform level of difficulty (typically uniformly low) so that, when given generous time limits, all testtakers should be able to complete all the test items correctly

Answer 66

speed test

Question 67

Could also be used to determine the number of items needed to attain a desired level of reliability that has the desired reliability

Answer 67

Spearman–Brown formula

Question 68

Widely used as a measure of reliability, in part because it requires only one administration of the test

Answer 68

coefficient alpha

Question 69

nternal consistency estimates of reliability, such as that obtained by use of the Spearman– Brown formula, are inappropriate for measuring the reliability of

Answer 69

heterogenous and speed tests

Question 70

If the variance of either variable in a correlational analysis is inflated by the sampling procedure, then the resulting correlation coefficient tends to be __

Answer 70

higher

Question 71

A statistic useful in describing sources of test score variability is the

Answer 71

variance

Question 72

This source of error fluctuates from one testing situation to another with no discernible pattern that would systematically raise or lower scores; increase or decrease test scores unpredictably

Answer 72

random error

Question 73

to evaluate the extent to which items on a scale relate to one another

Answer 73

internal consistency

Question 74

sources of error variance of alternate forms

Answer 74

test construction or administration

Question 75

simplest way of determining the degree of consistency among scorers in the scoring of a test

Answer 75

coefficient of inter-scorer reliability

Question 76

they influence test scores in a consistent direction; either consistently inflate scores or consistently deflate scores

Answer 76

systematic error

Question 77

test items or questions with three or more alternative responses, where only one is scored correct or scored as being consistent with a targeted trait or other construct

Answer 77

polytomous test items

Question 78

use: when assessing the stability of various personality traits

Answer 78

test-retest reliability

Question 79

obtained by correlating two pairs of scores obtained from equivalent halves of a single test administered once

Answer 79

split-half reliability

Question 80

When measuring something repeatedly, two influences interfere with accurate measurement:

Answer 80

(1) Time elapses between measurements (2) The act of measurement can alter what is being estimated

Question 81

when a time limit is long enough to allow testtakers to attempt all items, and if some items are so difficult that no testtaker is able to obtain a perfect score

Answer 81

power test

Question 82

an estimate of reliability obtained by correlating pairs of scores from the same people on two different administrations of the same test

Answer 82

test-retest reliability

Question 83

A statistical measure that can aid a test user in determining how large a difference should be before it is considered statistically significant

Answer 83

standard error of difference

Question 84

calculated to help answer questions about how similar sets of data are

Answer 84

coefficient alpha

Question 85

A test is said to be __ in items if it is functionally uniform throughout

Answer 85

homogeneous

Question 86

signifies the degree to which an item differentiates among people with higher or lower levels of the trait, ability, or whatever it is that is being measured

Answer 86

discrimination

Question 87

sources of error variance of test-retest

Answer 87

administration

Question 88

a value that according to CTT genuinely reflects an individual’s ability (or trait) level as measured by a particular test

Answer 88

true score

Question 89

It is a useful measure of reliability when it is impractical or undesirable to assess reliability with two tests or to administer a test twice (because of factors such as time or expense)

Answer 89

split-half reliability

Question 90

A range or band of test scores that is likely to contain the true score

Answer 90

confidence interval

Question 91

tied to the measurement instrument used

Answer 91

true score

Question 92

the inherent uncertainty associated with any measurement, even after care has been taken to minimize preventable mistake

Answer 92

measurement of error

Question 93

in homogenous test, it is reasonable to expect a high degree of

Answer 93

internal consistency

Question 94

The procedures of this provide a way to model the probability that a person with X ability will be able to perform at a level of Y

Answer 94

Item Response Theory (IRT)

Question 95

typically designed to be equivalent with respect to variables such as content and level of difficulty

Answer 95

alternate forms

Question 96

refers both to preventable mistakes and to aspects of measurement imprecision that are inevitable

Answer 96

error

Question 97

Reliable tests give scores that closely approximate __ scores

Answer 97

true score

Question 98

The degree of agreement or consistency between two or more scorers (or judges or raters) with regard to a particular measure

Answer 98

Inter-scorer reliability

Question 99

Of the three types of estimates of reliability, __ are perhaps the most compatible with domain sampling theory

Answer 99

measures of internal consistency

Question 100

Focuses on the degree of difference that exist between item scores

Answer 100

average proportion distance