Test Construction

Question 1

Classical Test Theory

Answer 1

*Assumes that obtained test scores are due to 1. true score variability and 2. measurement error.

Question 2

Reliability Coefficient

Answer 2

Range from 0 to 1.0
Amount of variability obtained test scores due to true score variability.

r= .80 (80% of score are due to true score variability and 20% due to measurement error).

.70 or higher is seen as minimally acceptable for most test

Question 3

Test-retest reliability

Answer 3

Provide info on the consistency of scores over time.

Administer test at baseline and then again later on. Correlating the 2 scores.

*Useful for test that are measuring characteristics that are stable over time.

Question 4

Alternate Forms Reliability

Answer 4

Provide info on the consistency of scores over different forms of the test and when second form is administered at a later time.

*When a measure has more than 1 form

Question 5

Internal Consistency Reliability

Answer 5

Provide info on the consistency of scores over different test items

*useful for test that measure a single content domain

Methods
-Coefficient Alpha (administering test to examinees and averaging the inter-item consistency)

Question 6

Inter-Rater Reliability

Answer 6

*subjectively test

Provide info on the consistency of scores assigned by different raters.

Methods
-Percent agreement
-Cohen Capa Coefficient

Question 7

Content Homogeneity

Answer 7

*Effect size of reliability coefficient

-Test that’s contents are homogeneous have larger reliability coefficients then those that are heterogeneous.

Question 8

Range of scores

Answer 8

*Effect size of reliability coefficient

-reliability coefficients are larger when tests scores are larger/ unrestricted in range.
-unrestricted range occurs when examinees included the sample are heterogeneous with regard to characteristics measured by the test (high, moderate and low)

Question 9

Guessing

Answer 9

*Effect size of reliability coefficient

-likelihood a test answer can be correctly answered by guessing, the lower the reliability coefficient.

Question 10

Reliability index

Answer 10

*Effect size of reliability coefficient

-theoretical correlation between true test scores and observed test scores.

*When the reliability coefficient is .81 the Reliability Index is the square root of .81 = .90

Question 11

Item Analysis

Answer 11

Used to determine which items to include in the test.

Used to determine item difficulty level and ability to discriminate between high and low total test scores.

Question 12

Item Difficulty

Answer 12

P = number of participants who answer correctly
- calculated by dividing the number of correct responses by the total number of responses.

value of P ranges from 0 to 1.0

*Moderately difficult items are preferred for most test. (.30 to .70)

Question 13

Item Discrimination

Answer 13

D Ranges from -1.0 to 1.0

Difference between examinees with high total test scores and percentage of examinees with low total test scores who also answered the question correctly

Question 14

Standard error of measurement & confidence intervals

Answer 14

Standard error of measurement is used to obtain a confidence interval.

-calculated by multiplying the test SD times the square root of 1 minus the reliability coefficient

68% confidence interval = add and subtract 1 standard error of measurement to and from the obtained score.

95% confidence interval = you add and subtract 2 standard error of measurement

99% confidence interval = you add and subtract 3 standard error of measurement

(an examinee obtained a score of 90 on a test that has a standard error of measurement of 5 and ask you to identify the 95% confidence interval for this score. To do so, you add and subtract 10 (two standard errors) to and from 90, which gives you a 95% confidence interval of 80 to 100. )

Question 15

Item Response Theory

Answer 15

Item based
Focuses on responses to individual test items.

*determine the probability of answering a test item correctly

*better suited for developing computerized tests.

Question 16

Content Validity

Answer 16

*Test that are designed to measure more than 1 behavior domains.

*Is it measuring all aspects of the domain.

Question 17

Construct Validity

Answer 17

*Test designed to measure a hypothetical trait (motivation, extraversion, ect).

*obtaining convergent and divergent validity.

Question 18

Convergent Validity

Answer 18

Degree to which scores on the test have high correlations with scores on other similar measures.

Question 19

Divergent Validity

Answer 19

Degree to which scores on the test have low correlation with scores on measures of unrelated constructs.

Question 20

Multitrait-Multimethod Matrix

Answer 20

a table of correlation coefficients that provide information about a test’s reliability and convergent and divergent validity.

Question 21

Concurrent Validity

Answer 21

*used to develop criterion validity

Obtaining scores on the predictor and criterion at the same time.

Question 22

Predictive Validity

Answer 22

Obtaining scores on the predictor before obtaining scores on the criterion.

Question 23

Criterion-Related Validity Coefficient

Answer 23

Ranges from -1 to +1
(closer to 1 the more accurate predictor scores will be at predicting criterion scores)

-Can be squared to determine the variability in one measure that is explained by or shared with the other measure.

Question 24

Sensitivity

Answer 24

the proportion of people with the disorder who are identified by the test as having the disorder. It’s calculated by dividing the true positives by the true positives plus the false negatives (TP/TP + FN).

Question 25

Specificity

Answer 25

the proportion of people without the disorder who are identified by the test as not having the disorder. It’s calculated by dividing the true negatives by the true negatives plus the false positives (TN/TN + FP).

Question 26

Hit Rate

Answer 26

the proportion of people who are correctly categorized by the test. It’s calculated by dividing the true positives and true negatives by the sample size (TP + TN)/(TP + TN + FP +FN)

Question 27

Z score

Answer 27

distribution has a mean of 0 and standard deviation of 1.0

(Subtracting raw score from the mean score and dividing by the SD)

M (raw score) = 110
X (mean score) = 100
SD = 5

110-100/5 = 2.0 (z score)

Question 28

T score

Answer 28

mean of 50 and standard deviation of 10. When an examinee’s raw score is equivalent to a T-score of 40, this means that his/her raw score is one standard deviation below the mean.

Question 29

Stanines

Answer 29

mean of 5 and standard deviation of 2 and range from 1 to 9.

Question 30

Percentile rank (PR) equivalents in a normal distribution

Answer 30

Percentile Rank of 2 = -2 SD
Percentile Rank of 16 = -1 SD
Percentile Rank of 84 = 1 SD
Percentile Rank of 98 = 2 SD