Ch. 5 - Reliability

Question 1

alternate forms

Answer 1

different versions of the same test or measure;

contrast with parallel forms

Question 2

alternate-forms reliability

Answer 2

estimate to the extent to which item sampling and other errors have affected scores on two versions of the same test;

contrast with parallel-forms reliability

Question 3

bias

Answer 3

a factor inherent within a test that systematically prevents accurate, impartial measurement

Question 4

classical test theory (CTT)

Answer 4

aka ‘true score theory / model’ …

system of assumptions about measurement that includes the notion that a test score (and even a response to an individual item) is composed of a relatively stable component that actually is what the test or individual item is designed to measure, as well as a component that is error.

Question 5

coefficient alpha

Answer 5

aka ‘Cronbach’s alpha’ and alpha…

a statistic widely employed in test construction and used to assist in deriving an estimate of reliability

more technically, equal to mean of all split-half reliabilities

Question 6

coefficient of equivalence

Answer 6

an estimate of parallel-forms reliability or alternate-forms reliability

Question 7

coefficient of generalizability

Answer 7

index of the influence that particular facets have on a test score

Question 8

coefficient of inter-scorer reliability

Answer 8

determines the degree of consistency among scorers in the scoring of a test

Question 9

coefficient of stability

Answer 9

estimate of test-retest reliability obtained during time intervals of six months or longer

Question 10

confidence interval

Answer 10

range or band of test scores that is likely to contain the “true score”

Question 11

content sampling

Answer 11

variety of the subject matter contained in the items;

aka item sampling, in context of variation between individual test items in a test or between test items in two or more tests

Question 12

criterion-referenced test

Answer 12

aka ‘domain-referenced testing’ and ‘content-referenced testing’

method of evaluation and a way of deriving meaning from test scores by evaluating an individual’s score with reference to a set standard (or criterion)

contrast with norm-referenced testing and assessment

Question 13

decision study

Answer 13

conducted at the conclusion of a generalizability study, this research is designed to explore the utility and value of test scores in making decisions

Question 14

dichotomous test item

Answer 14

test item or question that can be answered with only one of two response options (true/false, yes/no)

Question 15

discrimination

Answer 15

in IRT, degree to which an item differentiates among people with higher or lower levels of the trait, ability, or whatever it is that is being measured by a test

Question 16

domain sampling theory

Answer 16

a system of assumptions about measurement that includes the notion that a test score (and even response to an individual item) consists of a relatively stable component that actually is what the test or individual item is designed to measure as well as relatively unstable components that collectively can be accounted for as error

Question 17

dynamic characteristic

Answer 17

a trait, state, or ability presumed to be ever-changing as a function of situational and cognitive experiences

contrast with static characteristic

Question 18

error variance

Answer 18

in true score model…

component of variance attributable to random sources irrelevant to the trait or ability the test purports to measure in an observed score or distribution of scores

common sources of error variance include those related to test construction (including item or content sampling), test administration, and test scoring and intrepration

Question 19

estimate of inter-item consistency

Answer 19

an estimate of the reliability of a test obtained from a measure of inter-item consistency

Question 20

facet

Answer 20

in generalizability theory…

variables of interest in the universe including number of items in the test, amount of training the test scorers have had, purpose of the test administration, etc

Question 21

generalizability theory

Answer 21

aka domain sampling theory

system of assumptions about measurement that includes the notion that a test score (and response) consists of a relatively stable component that actually is what the test or individual item is designed to measure as well as relatively unstable components that collectively can be accounted for as error

Question 22

generalizability study

Answer 22

in context of generalizability theory…

research conducted to explore the impact of different facets of the universe on a test score

Question 23

heterogeneity

Answer 23

more generally, having diverse contents

heterogeneous test measures multiple factors

Question 24

homogeneity

Answer 24

describes degree to which a test measures a single trait

Question 25

inflation of range/variance

Answer 25

a reference to a phenomenon associated with reliability estimates wherein the variance of either variable in a correlational analysis is inflated by the sampling procedure used and so the resulting correlation coefficient tends to be higher contrast with restriction of range

Question 26

information function

Question 27

inter-item consistency

Answer 27

the consistency or homogeneity of the items of a test, estimated by techniques such as the split-half method

Question 28

internal consistency estimate of reliability

Answer 28

an estimate of the reliability of a test obtained from a measure of inter-item consistency

Question 29

inter-scorer reliability

Answer 29

aka inter-rater reliability, observer reliability, judge reliability, and scorer reliability an estimate of the degree of agreement of consistency between two and more scorers (or judges, raters, observers)

Question 30

item response theory (IRT)

Answer 30

aka latent-trait theory / model system of assumptions about measurement (including assumption that a trait being measured by a test is unidimensional) and the extent to which each test item measures the trait

Question 31

item sampling

Answer 31

aka content sampling variety of the subject matter contained in the items freq ref to in context of the variation between individual test items in a test or between test items in two or more tests

Question 32

latent-trait theory

Answer 32

aka latent-trait model system of assumptions about measurement, including the assumption that a trait being measured by a test is unidimensional, and the extent to which each test item measures the trait

Question 33

measurement error

Answer 33

refers to the inherent uncertainty associated with any measurement, even after care has been taken to minimize preventable mistakes

Question 34

odd-even reliability

Answer 34

estimate of split-half reliability of a test, obtained by assigning odd-numbered items to one half of the test and even-numbered items to the other half

Question 35

parallel forms

Answer 35

two or more versions of forms of the same test where, for each form, the means and variances of observed test scores are equal contrast with alternate forms

Question 36

parallel-forms reliability

Answer 36

as estimate of the extent to which item sampling and other errors have affected test scores on two versions of the same test when, for each form of the test, the means and variances of observed test scores are equal contrast with alternate-forms reliability

Question 37

polytomous test item

Answer 37

a test item or question with three or more alternative responses, where only one alternative is scored correct or scored as being consistent with a targeted trait or other construct

Question 38

power test

Answer 38

a test, usually of achievement or ability, which 1) either no time limit or such a long time limit that all test-takers can attempt all items and 2) some items so difficult that no test-taker can obtain a perfect score contrast with speed test

Question 39

random error

Answer 39

a source of error in measuring a targeted variable, caused by unpredictable fluctuations and inconsistencies of other variables in the measurement process contrast with systematic error

Question 40

Rasch model

Answer 40

reference to an IRT model with very specific assumptions about the underlying distribution

Question 41

reliability

Answer 41

the extent to which measurements are consistent or repeatable also extent to which measurements differ from occasion to occasion as a function of measurement error

Question 42

reliability coefficient

Answer 42

general term for an index of reliability or the ratio of true score variance on a test to the total variance

Question 43

replicability crisis

Answer 43

low replication rates commonly found in psychological research

Question 44

restriction of range/variance

Answer 44

aka restriction of variance phenomenon associated with reliability estimates wherein the variance of either variable in a correlational analysis is restricted by the sampling procedure used and so the resulting correlation coefficient tends to be lower contrast with inflation of range

Question 45

Spearman-Brown formula

Answer 45

equation used to estimate internal consistency reliability from a correlation of two halves of a test that has been lengthened or shortened inappropriate for use with heterogeneous tests or speed tests

Question 46

speed test

Answer 46

test usually of achievement or ability, with a time limit speed tests usually contain items of uniform difficulty level

Question 47

split-half reliability

Answer 47

estimate of the internal consistency of a test obtained by correlating two pairs of scores obtained from equivalent halves of a single test administered once

Question 48

standard error of a score

Answer 48

in true score theory, aka SEM a statistic designed to estimate the extent to which an observed score deviates from a true score

Question 49

standard error of measurement

Answer 49

(SEM, aka std err of score) in true score theory a statistic designed to estimate the extent to which an observed score deviates from a true score

Question 50

standard error of the difference

Answer 50

a statistic designed to aid in determining how large a difference between two scores should be before it is considered statistically significant

Question 51

static characteristic

Answer 51

a trait, state, or ability presumed to be relatively unchanging over time contrast with dynamic characteristic

Question 52

systematic error

Answer 52

a source of error in measuring a variable that is typically constant and proportionate to what is presumed to be the true value of the variable being measured contrast with random error

Question 53

test-retest reliability

Answer 53

estimate of reliability obtained by correlating pairs of scores from the same people on two different administrations of the same test

Question 54

transient error

Answer 54

source of error attributable to variations in the test-takers feelings, moods, or mental state over time

Question 55

true score

Answer 55

a value that, according to classical test theory, genuinely reflects an individual's ability (or trait) level as measured by a particular test

Question 56

true variance

Answer 56

in the true score model component of variance attributable to true differences in the ability or trait bring measured that are inherent in an observed score of distribution of scores

Question 57

universe

Answer 57

in generalizability theory the total context of a particular test situation, including all the factors that lead to an individual's test-taker's score

Question 58

universe score

Answer 58

in generalizability theory a test score corresponding to the particular universe being assessed or evaluated

Question 59

variance

Answer 59

a measurement of variability equal to the arithmetic mean of the squares of the differences between the scores in a distribution and their mean