Test Construction

Question 1

Alternate Forms Reliability

Answer 1

Alternative forms reliability is evaluated by administering two forms of the test to the same group of examinees and correlating the two sets of scores. When alternate forms are administered at about the same time, this form of reliability produces a coefficient of equivalence; when forms are administered at different times, it produces a coefficient of equivalence and stability. It is considered by some experts to be the best (most thorough) method for assessing reliability

Question 2

Classical Test Theory

Answer 2

Classical test theory describes observed variability in test scores as consisting of two components: true differences between examinees on the attribute(s) measured by the test and the effects of measurement (random) error. Reliability is a measure of true score variability.

Question 3

Coefficient Alpha/KR-20

Answer 3

Coefficient alpha and the Kuder-Richardson Formula 20 (KR-20) are used to assess internal consistency reliability and provide an index of average inter-item consistency. KR-20 can be used as a substitute for coefficient alpha when test items are scored dichotomously

Question 4

Construct Validity

Answer 4

Construct validity refers to the extent to which a test measures the hypothetical trait (construct) it is intended to measure. Methods for establishing construct validity include correlating test scores with scores on measures that do and do not measure the same trait (convergent and discriminant validity), conducting a factor analysis to assess the test’s factorial validity, determining if changes in test scores reflect expected developmental changes, and seeing if experimental manipulations have the expected impact on test scores

Question 5

Content Validity

Answer 5

Content validity refers to the extent to which a test adequately samples the domain of information, knowledge, or skill that it purports to measure. It is determined primarily by the judgment of subject matters experts and is important, for example, for achievement tests and job sample tests

Question 6

Convergent and discriminant validity

Answer 6

Convergent and discriminant validity are types of construct validity: When a test correlates highly with measure of the same and related constructs, this provides evidence of the test’s convergent validity. When a test has low correlations with measures of unrelated constructs, this provides evidence of its discriminant validity.

Question 7

Criterion contamination

Answer 7

Criterion contamination refers to bias introduced into a person’s criterion rating as a result of the rater’s knowledge about the person’s performance on the predictor. It tends to artificially inflate the correlation between scores on the predictor and criterion

Question 8

Criterion-referenced interpretations

Answer 8

Criterion-referenced interpretation involves interpreting a test score in terms of a prespecified standard (i.e. in terms of percent of content correct (percentage score) or by using a regression equation or expectancy table to predict performance on an external criterion based on score or status on a predictor.

Question 9

Criterion-related validity/concurrent and predictive

Answer 9

Criterion-related validity is important when predictor scores will be used to predict or estimate scores on a criterion (e.g. when selection test scores will be used to predict or estimate job performance ratings. It is evaluated by administering the predictor and criterion to a sample and correlating their scores to obtain a criterion-related validity coefficient. Criterion-related validity can be either concurrent (predictor and criterion scores obtained at about the same time) or predictive (predictor scores obtained before criterion scores)

Question 10

Cross-validation and shrinkage

Answer 10

Cross-validation is the process of re-assessing a test’s criterion-related validity on a new sample to check this generalizability of the original validity coefficient. Ordinarily, the validity coefficient “shrinks” (becomes smaller) on cross-validation because the chance factors operating in the original sample are not all present in the cross-validation sample.

Question 11

Factor analysis

Answer 11

Factor analysis is a multivariate statistical technique used to determine how many factors are needed to account for the intercorrelations among a set of tests, subtests, or test items. It can be used to assess a test’s construct validity by indicating the extent to which he test correlates with factors that it would and would not be expected to correlate with. Factors identified in a factor analysis can be either orthogonal (uncorrelated) or oblique (correlated)

Question 12

Factor loadings and communality

Answer 12

In a factor matrix, a factor loading is the correlation between a test (or other variable included in the analysis) and a factor and can be squared to determine the amount of variability in the test that is accounted for by the factor. The communality is the total amount of variability in scores on the test that is accounted for by the factor analysis (ie.g. by all of the identified factors).

Question 13

Incremental validity/true positives, false positives, true negatives, false negative

Answer 13

Incremental validity refers to the extent to which a predictor increases decision-making accuracy. It is calculated by subtracting the base rate from the positive hit rate. Terms to have linked with incremental validity are predictor and criterion cutoff scores, true and false positives, and true and false negatives.

True positives are people who scored high on the predictor (IV) and criterion (DV)

False positives scored high on the predictor (IV) but low on the criterion (DV)

True negatives scored low on the predictor (IV) and the criterion (DV)

False negatives scored low on the predictor (IV) but high on the criterion (DV)

Question 14

Item Characteristic Curve

Answer 14

When using item response theory to construct a test, an item characteristic curve (ICC) is derived for each item by plotting the proportion of examinees in the trout sample who answered the item correctly against the total test score, performance on an external criterion, or a mathematically derived estimate of a latent ability or trait.

Question 15

Item Difficulty

Answer 15

An item’s difficulty level is calculated by dividing the number of individuals in the trout sample who answered the item correctly by the total number of individuals. The item difficulty index (p) ranges in value from 0 (very difficult item) to 1.0 (very easy item). In general, an index of .50 is preferred because it maximizes differentiation between individuals with high and low ability and helps ensure a high reliability coefficient

Question 16

Item Discrimination

Answer 16

Item discrimination refers to the extent to which a test item discriminates between examinees who obtain high versus low scores on the entire test. The item discrimination index (D) ranges from -1.0 to +1.0. When all examinees in the upper-scoring group and none in the lower-scoring group answered the item correctly, D is + is +1.0 ; when none of the examinees in the upper-scoring group and all examinees in the lower-scoring group answered the item correctly, D equals -1.0

Question 17

Kappa Statistic

Answer 17

The kappa statistic is also known as the kappa coefficient and is used to evaluate inter-rater reliability when ratings represent a nominal or ordinal scale of measurement

Question 18

Multitrait-multimethod matrix

Answer 18

The multitrait-multimethod matrix is a table that is used to organize the correlation coefficients obtained when assessing a measure’s convergent and discriminant validity. Use of the matrix requires measuring at least two different traits using at least two different methods for each trait. Terms to have linked with the multi-trait-multi-method matrix are monotrait-monomethod, monotrait-heteromethod, heterotrait-monomethod, and heterotrait-heteromethod coefficients

Question 19

Norm-Referenced Interpretation

Answer 19

Norm-referenced interpretation involves interpreting an examinee’s test score in terms of the scores obtained by examinees in a normative (standardization) sample. Percentile ranks and standard scores (e.g. z-scores and T-scores) are types of norm-referenced scores.

Question 20

Orthogonal and Oblique Rotation

Answer 20

When conducting a factor analysis, the initial factors are rotated to simplify their interpretation. An orthogonal rotation produces uncorrelated factors, while an oblique rotation produces correlated factors.

Question 21

Relationship between reliability and validity

Answer 21

Reliability is a necessary but not sufficient condition for validity. In terms of criterion-related validity, the validity coefficient can be no greater than the square root of the product of the reliabilities of the predictor and criterion

Question 22

Reliability/Reliability coefficient

Answer 22

Reliability refers to the consistency of test scores (i.e. the extent to which a test measures an attribute without being affected by random fluctuations (measurement error) that produce inconsistencies over time, across items, or over different forms. Methods for establishing reliability include test-retest, alternative forms, split-half, coefficient alpha, and inter-rater. Most produce a reliability coefficient, which is interpreted directly as a measure of true score variability (e.g. a reliability coefficient of .80 indicates that 80% of variability in test scores is true score variability.

Question 23

Sensitivity and Specificity

Answer 23

Sensitivity and specificity provide information about a predictor’s accuracy when administered to a group of individuals who are known to have or not have the disorder (or other characteristic) of interest. Sensitivity is the percent of people in the tryout sample who have the disorder and were accurately identified by the predictor as having the disorder. Specificity is the percent of people in the tryout sample who do not have the disorder and were accurately identified by the predictor as not having the disorder.

Question 24

Split-half reliability/Spearman-Brown formula

Answer 24

Split-half reliability is a method of assessing internal consistency reliability and involves “splitting” the test in half (e.g. odd- versus even-numbered items) and correlating examinees’ scores on the two halves of the test. The split-half reliability coefficient tends to underestimate a test’s actual reliability and is usually corrected with the Spearman-Brown formula, which estimates what the test’s reliability would be if it were based on the full length of the test

Question 25

Standard error of estimate/confidence interval

Answer 25

The standard error of estimate (SEE) provides an index of error when predicting criterion scores from predictor scores and is used to construct a confidence interval around an examinee’s predicted criterion score. Its magnitude depends on two factors: the criterion’s standard deviation and the predictor’s criterion-related validity coefficient

Question 26

Test length/range of scores

Answer 26

Two ways to increase a test’s reliability are to increase the test’s length by adding items of similar content and quality and to increase the range of scores. The range of scores can be increased by increasing the heterogeneity of the sample in terms of the attribute(s) measured by the test and/or choosing items to include in the test so that the average difficulty level is in the mid-range (p= .50)

Question 27

Test-retest reliability

Answer 27

Test-retest reliability is evaluated by administering the same test to the same group of examinees on two different occasions and correlating the two sets of scores. It yields a coefficient of stability.