Stats - Test Construction

Question 1

For a test to be good, what two things does it need?

Answer 1

Reliability and validity

Question 2

Does validity or reliability involve consistency, repeatability and dependability in scores?

Answer 2

Reliability

Question 3

What’s the formula of True Score Model or Classical Test Theory (i.e., total variability = ______ + _______)

Answer 3

total variability = true score variability + error variability (a combo of truth & error)

Question 4

If true score variability is 80% then what percentage is error variability?

Answer 4

20%
total variability is always 100%

Question 5

If true score variability is 80% then what is the reliability score?

Answer 5

0.80 (always expressed as a coefficient)

Question 6

What’s the minimal acceptable reliability

Answer 6

0.80

Question 7

What are 3 common sources of error in tests, and what are they?

Answer 7

1. content sampling: items that do or don’t tap into a test-takers knowledge
2. time sampling: scores are different at two points in time due to factors related to the passage of time
3. test heterogeneity: when a test has heterogeneous items tapping more than one domain

Question 8

How’s reliability increased in relation to number of items?

Answer 8

Reliability increases when the number of items increases

Question 9

How’s reliability increased in relation to homogeneity of items?

Answer 9

the more homogeneous the items are, the greater the reliability

Question 10

How’s reliability increased in relation to range of scores?

Answer 10

An unrestricted range of scores increases the reliability

Question 11

Does the ability to guess increase or decrease reliability?

Answer 11

It decreases reliability

Question 12

What is test-retest reliability?

Answer 12

When you administer the identical test to the same sample of people at two points in time

Question 13

Coefficient of stability is another name for what?

Answer 13

Test-retest reliability

Question 14

What’s parallel forms reliability?

Answer 14

It’s calculated by the scores obtained by the same groups of people on two roughly equivalent but not identical forms of the same test administered at two different points in time

Question 15

Coefficient of equivalence is another name for what?

Answer 15

Parallel forms reliability

Question 16

What type of reliability is when you look at the consistency of the scores within the test; therefore, the test is administered only once to one group of people

Answer 16

Internal consistency reliability

Question 17

What are the two subtypes of internal consistency?

Answer 17

1. Split-half reliability
2. Kuder Richardson (KR-20/21) or Coefficient Alpha

Question 18

What 1 of 4 reliability estimates is associated with the Spearman brown prophecy formula?

Answer 18

Internal consistency reliability in the subtype of split-half reliability

Question 19

What is subtype called from internal consistency reliability where it finds the average of every possible way of splitting the test in half

Answer 19

Kuder Richardson or Coefficient Alpha

Question 20

What reliability estimate looks at the degree of agreement between two or more scorers when a test is subjectively scored?

Answer 20

Interrater (interscorer) reliability

Question 21

What reliability estimate connects to percent agreement, Pearson r, Kappa stats or Yule’s Y

Answer 21

Interrater reliability

Question 22

If the test was perfectly reliable what would be the standard error of measurement score?

Answer 22

0.0

Question 23

What is the highest score of the standard error of measurement?

Answer 23

It would be equal to the standard deviation of the test

Question 24

What are the 3 confidence intervals?

Answer 24

-1 & +1 = 68%
-2 & +2 = 95%
-3 & +3 = 99%

Question 25

What connects to the meaningfulness, usefulness or accuracy of a measure in terms of predicting? Is it a good predictor?

Answer 25

Validity

Question 26

Are the items on my test really measuring the right knowledge or skills to predict if someone's good for a job? Is this content, criterion or construct validity?

Answer 26

Content validity

Question 27

Whether the test is valid or accurate in terms of predicting? Can we use our test to predict something? (e.g., sales potential or salesmen on my team) Is this content, criterion, or construct validity?

Answer 27

Criterion validity

Question 28

Whether my instruments are measuring the trait that I think it's measuring (make a scale that measures assertiveness, but is it actually measuring assertiveness or is it measuring aggression? Is this content, criterion, or construct validity?

Answer 28

Construct validity

Question 29

What type of validity, content vs construct, measures skills/knowledge or measures a trait

Answer 29

Content= skills and knowledge Construct = always a trait

Question 30

What are the two subtypes of criterion-related validity?

Answer 30

1. concurrent (measuring the predictor & outcome at the same time) 2. predictive (delay between the measurement of predictor and the criterion)

Question 31

The criterion-related validity coefficient is Rxy; what does x represent, and what does y represent?

Answer 31

x = test/predictor scores y= outcome/criterion scores

Question 32

How do you find out how much the outcome (criterion variability) can be explained by the predictor if you have the criterion-related validity coefficient score (e.g., 0.50)

Answer 32

You square it (0.50 squared = 0.25 = 25%)

Question 33

What's the maximum value of the standard error of the estimate?

Answer 33

It's the same as the standard deviation of the criterion

Question 34

3 applications of the criterion related validity coefficient

Answer 34

1. Expectancy tables 2. Taylor Russel tables 3. Decision making theory

Question 35

Which of the 3 applications lists the probability that a person's criterion score will fall in a specific range based on the range in which that person's predictor score fell? 1. Expectancy Tables 2. Taylor-Russel Tables 3. Decision making theory

Answer 35

1. Expectancy Tables

Question 36

Which of the 3 applications numerically shows how much more accurate selection decisions are when a particular predictor test as opposed to not using one? 1. Expectancy Tables 2. Taylor-Russel Tables 3. Decision making theory

Answer 36

2. Taylor Russel Tables

Question 37

What's the definition of a base rate? (Taylor Russel Table)

Answer 37

the rate of selecting successful employees without using a predictor test

Question 38

What's the definition of a selection ratio? (Taylor Russel Table)

Answer 38

the proportion of available openings to available applicants (e.g., 3 openings with 10 applicants)

Question 39

What's the definition of incremental validity (Taylor Russel Table)

Answer 39

it's the amount of improvement in the success rate that results from using a predictor test e.g., base rate = 0.40 (40%) to 0.65 (65%) incremental validity = 65% - 45% = 25% incremental validity = 25%

Question 40

When is incremental validity optimized? (in connection to base rate & selection ratio)

Answer 40

1. Moderate base rate (around 0.5) 2. Low selection ratio (around 0.1)

Question 41

Which of the 3 applications takes performance predictions based on predictor tests and compares them with the actual criterion outcome? 1. Expectancy Tables 2. Taylor-Russel Tables 3. Decision-making theory

Answer 41

3. Decision-making theory

Question 42

Draw out a decision-making theory table: where does criterion/outcome go, where does predictor/test go, & where do false negative, true negative, false positive & true positive go?

Answer 42

Criterion/outcome = Y (up & down axis) Predictor/test = X (right to left axis) false negative = top left true negative = bottom left false positive = bottom right true positive = top right

Question 43

If a company wanted to reduce false positives of a pregnancy test, what do you do to the predictor & criterion cut offs? 1. Increase both 2. Decrease both 3. Increase predictor & decrease criterion 4. Decrease predictor & increase criterion

Answer 43

3. Increase predictor & decrease criterion i.e., make the false positive box smaller in the decision-making theory table

Question 44

What does cross-validation do to the criterion-related validity coefficient?

Answer 44

Results in shrinkage of the criterion-related validity coefficient

Question 45

What does correction for attenuation calculate concerning validity and the reliability of predictor (x) & criterion (Y)?

Answer 45

How much higher validity would be if the predictor (X) and criterion (Y) were both perfectly reliable

Question 46

What's it called when the criterion is subjectively scored, and the rater knows how people did on the predictor before assigning them to the criterion: (e.g., if teacher knows student's IQ scores before an assignment and it and then they give students with higher IQs higher grades on the assignment)

Answer 46

Criterion Contamination

Question 47

What does criterion contamination result in? (in connection to criterion-related validity coefficient scores)

Answer 47

inflated/spuriously high criterion-related validity coefficient

Question 48

What type of validity looks at how adequately a new test measures a trait?

Answer 48

Construct validity

Question 49

Two subtypes of construct validity

Answer 49

1. Convergent validity - come together 2. Divergent (discriminant) validity - go apart

Question 50

With convergent vs divergent validity, which measures the same trait and which measures different traits?

Answer 50

Convergent = comparing measures of the same trait Divergent = comparing measures of different traits

Question 51

With convergent vs divergent validity, which one wants high correlations, and which one wants low correlations?

Answer 51

Convergent = high correlations Divergent = low correlations

Question 52

With convergent vs divergent validity, which relates to high correlations of monotrait-heteromethod (same traits are measured by different methods) and which relates to low correlations of heterotrait-monomethod (dif traits are measured by same method)

Answer 52

1. Convergent = high correlations of monotrait-heteromethod 2. Divergent = heterotrait-monomethod

Question 53

With standard error of measurement, where do you plot the person's score in relation to a bell shaped distribution and how do you figure out what the other numbers are in relation to +1, +2, +3, -1, -2, -3

Answer 53

The person's score goes in the middle (0) of the distribution & then add or subtract the standard error of measurement to the person's score (e.g. person's score = 70 & standard error of measurement = 5, then -1 would be 65 and +1 would be 75)

Question 54

Cronbach's Alpha is a measure of...

Answer 54

Internal Consistency Reliability

Question 55

Reliability is always highest when test items are (heterogenous/homogenous), subjects are (heterogenous/homogenous) and the number of items are (high or low)

Answer 55

test items = homogenous subjects = heterogenous # of items = high