Chapter 11 - Assessing Psychometric Quality

Question 1

Item Analysis

Answer 1

• How developers evaluate the performance of each test item.

Question 2

Quantitative Item Analysis

Answer 2

• Statistical analyses of the responses test takers gave to individual items.

Question 3

Item Difficulty

Answer 3

• The percentage of test takers who respond correctly.
• We calculate each item’s difficulty or p-value by dividing the number of persons who answered correctly by the total number of persons who responded to the questions.
• We get this information for the pilot test.
• They disregard questions that are too hard or too easy.

Question 4

Discrimination Index

Answer 4

• Compares the performance of those who obtained very high test scores with the performance of those who obtained very low test scores.

Question 5

Item-Total Correlation

Answer 5

• Another way to assess the ability of individual test items to discriminate high-scoring individuals from lower-scoring ones.
• This is a measure of the strength and direction of the relationship between the way test takers responded to one item and the way they responded to all of the items as a whole.

Question 6

Interitem Correlation Matrix

Answer 6

• Displays the correlation of each item with every other item.
• Usually, each item has been coded as a dichotomous variable - correct 1 or incorrect 0.

Question 7

Phi Coefficients

Answer 7

• Result of correct two dichotomous variables.

Question 7

Empirically Based Tests

Answer 7

• Tests designed so that test scores can be used to sort individuals into two or more categories based on their scores on the criterion measure.

Question 8

Subtle Questions

Answer 8

• Questions that have no apparent relation to the criterion.

Question 9

Item Response Theory (IRT)

Answer 9

• This theory relates the performance of each item to a statistical estimate of the test taker’s ability on the construct being measured.
• A measure of the relationship between an individual’s performance on one test item and the test takers’ levels of performance on the overall measure of the construct the test is measuring.

Question 10

Item Characteristic Curves (ICCs)

Answer 10

• The line that results when we graph the probability of answering an item correctly with the level of ability on the construct being measured.
• The ICC provides a picture of the item’s difficulty and how well it discriminates high performers from lower performers.

Question 11

Computerized Adaptive Testing (CAT)

Answer 11

• All test takers start with the same small set of questions.
• As the test progresses, the computer software chooses and presents each test taker with harder or easier questions depending on how well the test taker answered previous questions.

Question 12

Item Bias

Answer 12

• When an item is easier for one group than for another group.

Question 13

Acculturation

Answer 13

• The degree to which an immigrant or a minority member has adapted to a country’s mainstream culture.

Question 14

Qualitative Item Analysis

Answer 14

• Non-statistical means of evaluating qualitative data which normally refers to the analysis of text.
• Used when qualitative analysis procedures are used when test developers ask test takers for verbal or written feedback about test questions.

Question 15

Construct Bias

Answer 15

• Arises when items do not have the same meaning from culture to culture or subculture.

Question 16

Method Bias

Answer 16

• Arises when the mechanics of the test work differently for various cultural groups.

Question 17

Differential Item Functioning

Answer 17

• Arises when test takers from different cultures have the same ability level on the test construct, but the item or test yields very different scores for the two cultures.

Question 18

Why is revision important in test development?

Answer 18

• They use different kinds of analysis in order to pick the best questions to fulfill their goal.

Question 19

How are the final items chosen?

Answer 19

• Choosing the items that make up the final test requires the test developer to weigh each item’s evidence of validity, item difficulty and discrimination, interitem correlation, and bias.

Question 20

What is the first part of the validation process?

Answer 20

• Establishing evidence of validity based on test content.
• Carried out as the test is developed.

Question 21

Generalizable

Answer 21

• Meaning the test can be expected to produce similar results even though it has been administered in different locations.

Question 22

Replication

Answer 22

• The process of replication involves a final round of test administration to another sample of test takers representative of the target audience.

Question 23

Cross-Validation

Answer 23

• This process breaks the original sample used in the original validation study into two parts.
• This can be done without having to administer the test to a second group of test takers.

Question 24

Measurement Bias

Answer 24

• When the scores on a test taken by different subgroups in the population need to be interpreted differently because of some characteristic of the test not related to the construct being measured.

Question 25

Predictive Bias

Answer 25

• Occurs when the predictions made about a criterion score based on a test score are different for subsets of test-takers.

Question 26

Differential Validity

Answer 26

• When a test yields significantly different validity coefficients for subgroups.

Question 27

Single-Group Validity

Answer 27

• Test is valid for one group but not for another group.

Question 28

Slope Bias

Answer 28

• It occurs when the slopes of the separate regression lines that relate to the predictor to the criterion are not the same for one group as another.

Question 29

Why is test fairness important?

Answer 29

• Psych tests are used to compare individuals, their purpose is to identify or illuminate the differences among individuals.
• The results of differences should be based on the trait or characteristic being measured.

Question 30

Accessibility

Answer 30

• Pertains to the opportunity test takers have to demonstrate their standing on the constructs the test is designed to measure.

Question 31

Universal Design

Answer 31

• The idea behind universal design is that tests should be constructed from the outset in such a way that accessibility is maximized for all individuals who may take the test in the future

Question 32

Cut Scores

Answer 32

• Decision points for dividing test scores into pass-fail groupings.

Question 33

What is the purpose of test norms?

Answer 33

• To provide a reference point or structure for understanding one person’s score.

Question 34

Subgroup Norms

Answer 34

• Statistics that describe subgroups of the target audience.