Lecture 6: Essential Test Item Consideration

Question 1

Test items

Answer 1

the units that make up a test and the means through which samples of test taker behaviors are gathered

Question 2

Item Analysis

Answer 2

term that refers to all techniques used to assess the characteristics of test items and evaluate their quality during the process of test development and test construction

Question 3

Qualitative item analysis

Answer 3

rely on judegments of reviewers concerning the substantive and stylistic charcateristics of items as well as their accuracy and fairness

Question 4

Qualitative criteria (how do you evaluate criteria qulitatively?)

Answer 4

• appropriateness of item content and format
• clarity of expression
• grammatical correctness
• adherence to “some basic rules for writing items that have evolved over time”

Question 5

Quantitative item analysis

Answer 5

involves a variety of statistical procedures designed to ascertain the psychometric characteristics of items based in the responses obtained from the samples used in the process of test development

Question 6

Bias

Answer 6

any systematic error that enters into scores and affects their meaning in relation to what scores are designed to measure or predict

Question 7

The context of item analysis

Answer 7

• usage of simple statistical procedures
• information on item behavior
• practical features of interest

Question 8

When is the decision to create a test developed?

Answer 8

when developer realized that either no test exists for a particular purpose or that the existing tests for certain purpose are not adequate for a reason

Question 9

Planning a test entails specifying:

Answer 9

• the contruct or knowledge domains that the test will asess
• the typoe of population with which the test will be used
• the objectives of the items to be developed
• the concrete means through which teh behavior samples will begathered and scored

Question 10

Steps in the Test development Process

Answer 10

1. Item generation
2. Qualitative item analysis
3. Revision/replacement of items
4. Pilot study
5. Evaluation of pilot study results
6. Potentil modification of items
7. Additional pilot studies
8. Determination and fixation of test length
9. Test norming
10. Test publication

Question 11

Item generation

Answer 11

by writing (or otherwise) creating the test items, the administration and scoring procedures

Question 12

Qualitative item analysis

Answer 12

• submitting the pool item to experts
• to identify items that may be a disadvantage, or be offensive to any particular demographic group for which the test is intended

Question 13

Revision/Replacement of items

Answer 13

items that are identified by the reviwers as inadequate or problematic from the point of view of subject matter, offensiveness, or bias

Question 14

Pilot study

Answer 14

tryout out items that have been gathered and reviewed on samples

Question 15

Evaluation of pilot study results

Answer 15

through quantitative item analysis and additional qualitative analysis

Question 16

Potential modification of items

Answer 16

adding, deleting or modifying test items as needed

Question 17

Additional pilot studies

Answer 17

cross-validation: for checking wthether item statistics remain stable across different groups until a satisfactory set of items is obtained

Question 18

Determination and fixing of test length

Answer 18

• and the sequencing of items
• administration and scoring procedures that will standard in the final from of the test - on the basis of the foregoing analysis

Question 19

Test norming

Answer 19

administering the test to a new sample of individuals in order to develop normative data or performance criteria, etc.

Question 20

Test publication

Answer 20

Publishing the final form, along with administration and scoring manual
(accompanying documentation of the use for which the test is intended)

Question 21

computer adaptive testing (CAT)

Answer 21

relies on banking pools of items that have been carefully calibrated with respoect to the information they convey

Question 22

Types of responses they require from test takers (two types)

Answer 22

1. selected-response items

2. constructed-response items

Question 23

Selected-response items

Answer 23

• objective or fixed-response items
• present a limited number of response alternatives (from which test takers must choose)
• Pass/fail items (dichotomous), ranking scale items (polytomous)

Question 24

Forced choice

Answer 24

objective items that require test takers to chosse between two or more alternatives is most or least characteristic of themselves
- mainly used in multidemensional personality inventories

Question 25

Ipsative scores

Answer 25

ordinal numbers that simply reflect test taker's raking of constructs assesses by the scaled within a forced-choice format test

Question 26

Advantages of Selected-Response items

Answer 26

- easy to administer, easy to score - high degree of objectivity - enhance test score reliability - group administration

Question 27

Disadvantage of Selected-Response items

Answer 27

- restrict the potential responses to a selection chosen by test developers - susceptible to guessing (pass/fail) - response styles (tendiency towards middle/extremes) - responses can be disorted by social desirability, self-monitoring, etc.

Question 28

Constructed- response items

Answer 28

- "Free" response format - open-ended (variety is limitless) - but usually some constarints on response behavior in instructions (time limit, length of response, usage of materials, etc.)

Question 29

In personality testing

Answer 29

the use of constructed responses is limited mainly to projective techniques also know as performance-based measures of personality - respond as freely as possible and reveal aspect of personality

Question 30

Advantages of Constructed-response items

Answer 30

- do not restrict response behavior to pre-selected options - may elicit greater acceptance of a test - provide richer samples of the behavior of examinees (unique charcateristic can emerge)

Question 31

Disadvantages of Contructed-response items

Answer 31

- less objective scoring (might be evaluated different by different scorers) - highly diverse responses, questioning their comparability - test lenght: responses require more time for administration and scoring

Question 32

What is the most important aspect in qualitative item analysis?

Answer 32

- item validity: does a specific item carry its own weight within a test by eliciting information that advances the purpose of the test? (also item discrimination)

Question 33

Classical methods of item analysis

Answer 33

1. Item difficulty | 2. Item Discrimination (item validity)

Question 34

Qualitative evaluation (inspect content regarding...)

Answer 34

- approprateness - difficulty level - possible bias or offensiveness toward any group

Question 35

Quantitative evaluation

Answer 35

of item diffulty is carried out through statistcis that assess wtather items perform the way they were intended to perform

Question 36

Item difficulty

Answer 36

- diffculty level of a test as a whole is a function of the difficulty levels of the indivudal items that make up a test (easy items = easy test) - item diffculty is sample dependent (depends on the ability of test takers)

Question 37

proportion (or percentage) passing (p)

Answer 37

the higher the prcentage passing, the easier the item is | - when normally distributed, p values can be trasnformed into z-values

Question 38

Z- values

Answer 38

relative difficulty levels of items can be compared across various groups by administering a commong set of item (anchor items)

Question 39

How can item diffulty be gauged?

Answer 39

- difficulty in words in the frequency with which they are used in the language - quantitaive indexes: percentage of test takers who aswer an item correctly (p-value)

Question 40

Absolute scaling

Answer 40

allows for the difficulty items to be placed on a uniform numerical scale for samples of test takers at different ability levels

Question 41

Item difficulty levels, test difficulty levels, and test purpose

Answer 41

- the average score on a test is the same as the average diffuclty on its items - average percenatge passing (p) for the items in a test id 80%, the average score on the test will be 80% as well

Question 42

insufficient ceiling

Answer 42

when test items are too easy for a certain group | - the distribution will be negatively skewed

Question 43

inadequate floor

Answer 43

when the test items are too difficult for a certain group | - the score distribution is positively skewed

Question 44

Distractors

Answer 44

- have a great deal of influence on item difficulty - the number of sitractors directly affects indexes of item difficulty because the probability of guessing correctly is higher when the number pf choice is smaller

Question 45

Item validity - Item discrimination

Answer 45

the extent to which items elicit responses that accurately differentiate test takers in terms of behavior, knowledge, or other charcateristics that a test is designed to evaluate

Question 46

Item validation criteria

Answer 46

- Internal criteria - total test score is used to validate item (homogeneity of the test increases) - > the reliability indexes based on interitem consistency is enhanced - external criteria- test are used in valdating items - > the validity of scores on the test as a whole is enhanced

Question 47

unidimensional traits

Answer 47

total score may be used to validta items | - all test items should correlate highly with the total score and each other

Question 48

Complex and multifaceted constructs

Answer 48

- items are validated against external criteria that are also mor global - not necessarily have to correlate highly with one another (not homogenous?)

Question 49

Item Validity statistics

Answer 49

all statistical procedures used to gauge the degree to which items discriminate in term of criterion require info about... 1. item performance 2. criterion stading for individuals in the samples from which the item discrimination statistics are extracted

Question 50

Index discrimination (D)

Answer 50

the differences in the percenatge or proportion of test takers in the upper and lower criterion groups who pass a given item or answer in the keyed direction - positive discrimination indexes: more individuals in the upper criterion group

Question 51

Computation of D

Answer 51

test takers must be classified into distinct criterion groups based either on their total scores on the test or on some external indicator of their standing on the constructs assessed - once the groups are created, the percentage (p) of individuals within each group who üasses the item is calculated

Question 52

Correlation coefficients

Answer 52

- test theory method used for expressing the relationship between performance on an item and criterion

Question 53

Point biserial (rpb) correlation coefficient

Answer 53

when item scores are dichotomous (pass/fail) and the criterion measure is continuous

Question 54

phi coefficient

Answer 54

- when item scores and the critetion measure are both dichotomous - both can range from -1 to +1

Question 55

Test can be classified into three categories

Answer 55

1. Pure speed tests 2. Pure power tests 3. Blend of speed and power

Question 56

Pure speed test

Answer 56

difficulty is manipulated mainly through timing, limits are so short that most test takers connot complete all the items - when test takers fisnih all items - actual capacity has not been determined

Question 57

Pure power test

Answer 57

- have no time limits | - difficulty is mainly manipulated by increasing or decreasing the complexity of items

Question 58

Blend of speed and power

Answer 58

most ability tests - fall between the extremes of pure-speed and pure-power continuum (time limits allow test takers to attempt all items)

Question 59

Item-test regression

Answer 59

necessary to calculate the proportion of individuals at each total score who passed a given item (combine info about item difficulty and item discrimination)