Lecture 3 Test Development (Catherine)

Question 1

What are the 5 stages of Test Development?

Answer 1

1. Test Conceptualisation
2. Test Construction
3. Test Try Out
4. Item Analysis
5. Test Revision

Question 2

What is the aim of Test Conceptualisation?

Answer 2

To establish the reasons for designing or revising a psychological test

Question 3

What is the aim of Test Construction?

Answer 3

To ascertain how numbers should be assigned to psychological attributes and which scales of measurement should be used

Question 4

What is the purpose of the Test Try Out phase?

Answer 4

To consider to whom the test should be administered and to generate data about its reliability and validity

Question 5

What is the purpose of the Item Analysis phase?

Answer 5

• To identify the properties of the test’s items & scales.
• to ascertain which types of analyses need to be performed to understand item validity, reliability, difficulty and discrimination.
• in short, to identify which are ‘good’ items and which are ‘bad’ items

Question 6

What is the purpose of Test Revision?

Answer 6

Determining the items of the test that may need to be revised or discarded.
*establishing whether the test is measuring what it was designed to measure

Question 7

What are the key stages involved in test conceptualisation?

Answer 7

• Conceiving the need for a test
• Considering the assumptions for test design
• Running a Pilot Test
• Conceptualising a Measure

Question 8

What are the key considerations when Conceiving the need for a test?

Answer 8

• There is always a question in the mind of the test developer e.g.
• There ought to be an instrument designed to measure [some psychological construct] in [such & such] a way
• even if a measure does exist, it may have poor psychometric properties (e.g. low reliability, poor construct validity and suspect content validity)

Question 9

It what way do the key questions a test developer poses when designing a test correspond with assumption 1 of test design?

Answer 9

Assumption 1 Psychological states & traits exist:

1. What is the test designed to measure?
2. Is there a need for this test?
3. What content will this test cover?

Question 10

It what way do the key questions a test developer poses when designing a test correspond with assumption 2 of test design?

Answer 10

Assumption 2 Psychological states & traits can be Quantified & Measured

4. What is the ideal format for this test?
5. What type of responses will be required of test takers?

Question 11

It what way do the key questions a test developer poses when designing a test correspond with assumption 3 of test design?

Answer 11

Assumption 3 Test related behaviour predicts non-test related behaviour

6. Should more than one form of the test be developed?
7. How will meaning be attributed to scores on this test?

Question 12

It what way do the key questions a test developer poses when designing a test correspond with assumption 4 of test design?

Answer 12

Assumption 4 Psychological tests have strengths & weaknesses

8. Who benefits from administration of this test?
9. Is there any potential for harm as a result of the administration of the test?

Question 13

What are the key considerations in the stage of pilot work?

Answer 13

• Pilot work refers to the preliminary work surrounding the creation of a prototype of a test
• During this phase a test developer attempts to determine how best to measure a targeted construct using a number of strategies: e.g. literature review, experimentation
• Test items may be pilot studies to evaluate whether they should be included in the final form of the instrument. (creation, revision, deletion of test items)

Question 14

What are the 4 key considerations when conceptualising a measure to be used in a test?

Answer 14

• What is the test designed to measure?
• Is there a need for this test?
• What content will the test cover?
• What is the ideal format of the test?

Question 15

What are the key elements required for test construction?

Answer 15

*Identifying appropriate measurement scales (N.O.I.R.)
*Developing a scale appropriate for the test: e.g.
-a summative scale,
-binary rating scale
-faces scale
-Method of paired comparisons
-comparative scaling
Categorical scaling

Question 16

Describe the key aspects of a summative scale and give examples

Answer 16

• a summative scale is a scale on which a test-taker’s answers to test/scale answers may be summed to create an aggregate score
• This aggregate score is thought to indicate the strength of the test takers traits attitude or other characteristic
• A Likert scale is a popular summative scale
• Test developers often treat summative scales as interval even though they are actually ordinal

Question 17

Describe the key aspects of a Likert (summative) scale

Answer 17

A Likert Scale typically provides the test taker with 5 or 7 possible responses along an ‘agree’ to ‘disagree’ continuum
*Easy to construct and are used extensively in psychology because it yields ordinal level data, which approximates interval-level data well enough for data analytic purposes

Question 18

What are the key points of the Method of paired comparisons

Answer 18

*Test-taker has to choose one of two options (e.g. a statement, object, picture) on the basis of some rule
*The value (e.g. 1 or 0) of each option in each paired comparison is determined by judges prior to test administration
*test takers must select the behaviour they feel is most justified
(NB Ordinal Data)

Question 19

What are the key points of Comparative Scaling?

Answer 19

*a test taker must sort or rank stimuli according to some rule e.g. best to worst, most justifiable to least justifiable
(NB Ordinal Data)

Question 20

What are the key points of Categorical Scaling?

Answer 20

Test takers must categorise stimuli according to some rule
e.g. sort the behaviours into good sleep practice and bad sleep practice
(NB can give nominal or ordinal data)

Question 21

What are the key points to consider when Writing Test Items?

Answer 21

• The first step in writing items is to create an item pool or item bank
• as a general rule it is advisable to have at least twice as many items as you will include in your final draft test/measure
• the process of test try out, item analysis, and test revision, will knock many of these items out as the test developer strives to maximise reliability & validity

Question 22

What types of item format are there that can be used in writing test items?

Answer 22

1. Constructed response format
   * completion answer
   * short answer
   * essay
2. Selected response format
   * matching choice
   * true-false
   * multiple-choice

Question 23

What are the key points from Completion items from the constructed response format of test items?

Answer 23

Completion items:

• Test-taker provides a word or phrase that completes a sentence
• There must be a specific correct answer
• Poorly designed completion items can lead to scoring problems

Question 24

What are the key points from Short Answer & Essay items from the constructed response format of test items?

Answer 24

• Short Answer items:
• a word, a sentence, & paragraph all qualify as a short answer
• If more than 2 paragraphs, then the item is more likely to be an essay item
• Essay items:
• Useful when the examiner wants the test-taker to demonstrate in-depth knowledge
• Allows for creative integration & expression of what has been learned
• Only a limited range of material can be examined this way

Question 25

What are the key points from Matching Choice items from the Selected Response format of test items?

Answer 25

Matching items:

The test taker is presented with premises and responses and must match the correct ones to each other

Question 26

What are the key points from True-false items from the Selected Response format of test items?

Answer 26

*True-False items are also known as Binary choice items
*Must contain a single unambiguous idea and be short
Expectancy tables are used in evaluating criterion-related validity
*include: yes-no, agree-disagree, true-false

Question 27

What are the key points from Multiple Choice items from the Selected Response format of test items?

Answer 27

• Multiple choice items have 3 elements:
• A stem
• A correct option
• Multiple incorrect alternative options (distractors, foils)
• There must be one correct answer and all options must be plausible and grammatically comparable

Question 28

What is used to evaluate criterion-related validity

Answer 28

Expectancy tables are used in evaluating criterion-related validity

Question 29

What are the key points from Complex Multiple Choice items from the Selected Response format of test items?

Answer 29

Complex multiple-choice formats include:

• Classification,
• If-Then Conditions
• Multiple conditions
• Oddity

Question 30

What are the key points from Classification in Complex Multiple Choice items from the Selected Response format of test items?

Answer 30

The test taker classifies a person, object or condition into one of several categories designed in the stem:
e.g. Jean Piaget is best described as a ….. Psychologist:
a, Clinical, b Development, c Psychometric, d Social (b)

Question 31

What are the key points from If-Then conditions in Complex Multiple Choice items from the Selected Response format of test items?

Answer 31

The test taker must decide the consequence of one or more condition being present
e.g. If the true variance of a test increases but the error variance remains constant, which of the following will occur?
a. reliability will increase
b. reliability will decrease
c. observed variance will decrease
d neither reliability nor observed variance will change
(a)

Question 32

What are the key points from Multiple conditions in Complex Multiple Choice items from the Selected Response format of test items?

Answer 32

The test taker uses two or more conditions in the statements to draw a conclusion:
Given Mary’s raw score on a test is 60, the test mean is 59, and the standard deviation is 2, what is Mary’s z score?
a. -2.00, b -.50, c .50, d 2.00
(c)

Question 33

What are the key points from Multiple true-false conditions in Complex Multiple Choice items from the Selected Response format of test items?

Answer 33

The test taker decides whether one, all, or none of the two or more statements or conditions listed in the stem are correct:

Is it true that (1) Alfred Binet was the father of intelligence testing, & that (2) his first intelligence test was published in 1916?
a. Both 1 & 2
b. 1 but not 2
c. 2 but not 1
d. Neither 1 or 2
(c)

Question 34

What are the key points from Oddity conditions in Complex Multiple Choice items from the Selected Response format of test items?

Answer 34

The test taker indicates which option does not belong with the others:

which of the following names does not belong with the others?

a. Alfred Adler
b. Sigmund Freud
c. Carl Jung
d. Carl Rogers
(a) Adler is not technically a personality theorist;Adler also confronted his patients rather than building rapport with them