Lecture 3 - Test Development 1: part2 - Construction (DN)

Question 1

What is the difference between scaling & measurement?

Answer 1

• Measurement: the assigning of numbers to some attribute
• Scaling: the method (mechanism) by which we assign numbers (rule setting)

Question 2

What types of rules guide measurement?

Answer 2

• Nominal rule
  
  • classifies only
• Ordinal rule
  
  • slightly more sophisticated
  • it is ordered
  • train example - order says nothing about distances between stations
  • has both nominal & ordinal properties
• Interval rule
  
  • has nominal & ordinal properties
  • equal distance between
  • no absolute zero
  • zero is just arbitrary
  • e.g., celsius (zero = freezing, but not the absence of tempretaure)
• Ratio rule
  
  • true zero point
  • not a lot in psych testing

57:00

Question 3

Which rule would apply if the following exist

• Categories
• Ordinal
• Equal Intervals
• Absolute Zero Point

Answer 3

Ratio Rule

Question 4

Which rule would apply when the following exists

• Categories

Answer 4

Nominal rule

Question 5

Which rule would apply if the following exist

• Categories
• Ordered (ordinal)
• Equal Intervals

Answer 5

Interval rule

Question 6

Which rule would apply when the following are present

• Categories
• Ordering

Answer 6

Ordinal rule

Question 7

What level of scale would you require if you are wanting to get an aggregate (summative) score?

i.e., Summative Scale

Answer 7

• At least Interval
• 1:09*

Question 8

What level of data is produced when using a likert scale?

1:10:30

Answer 8

• ordinal-level data
• because they do not actually have equal levels between responses
  
  • however they are treated as interval (summative) scale
  • because they behave interval like (although theoretically there are not equal distances between strongly disagree & disagree)

Question 9

What level of data is derived from binary choice scale?

Answer 9

• nominal
• not as much info as a likert, but its simpler

1:13:25

Question 10

Why would you use a faces scale?

Answer 10

• good for children
• they dont understand things in the same way adults do

1:14:20

Question 11

What are some other scaling methods that yield ordinal data?

1:17

Answer 11

• method of paired comparisons (1:17:19)
  
  • must choose one on basis of some rule
• comparative scaling (1:19:19)
  
  • sorting/ranking along a dimension
  • what is worse cheating on taxes or cheating on test
• categorical scaling (can be nominal data)
  
  • sorting stimuli according to rule
  • good or bad / best or worst
  • essential - not essential (our questionnaire)
  • depends on how question is framed as to whether you get ordinal or nominal data (with categorical)

Question 12

What are the main item formats used when writing test items?

1:27:10

Answer 12

• ** Constructed-response** format
  
  • 1. Completion item
  • 2. Short-answer
  • 3. Essay
• Selected-response format
  
  • 1. Matching items
  • 2. True-False
  • 3. Multiple choice

Question 13

What are the different types of constructed-response format?

Answer 13

1. Completion item (1:25)

• provides word to complete sentence
• must be a specific correct answer
• poor design can lead to scoring problems

2. Short-answer (1:31:15)
   * word, sentence, paragraph in answer to a highly specific question
3. Essay (132:10)

Question 14

What are the different types of selected-response formats?

Answer 14

1. Matching
   2. True-False (1:34:20)

• must contain a single, unambigous idea & be short
• use kuder-richardson for binary

3. Multiple choice (1:35:35)

• a stem
• a correct item
• multiple incorrect alternative options (distractors, foils)
• stem - expectancy tables are used to evaluate this
  
  • could use kuser-richardson & look at it internally
  • or look at all 4 and use chronbach alpha

Question 15

What are the four complex multiple choice formats?

1:38:45

Answer 15

• Classification
• If-then conditions
• Multiple conditions
• Multiple true-false
• Oddity

Question 16

Which scaling methods did we use & why?

1:23:20

Answer 16

Previous research (Talbot et. al.)

• used Likert (ordinal)
• made it summative (assumed interval) to get a composite score

We will use

• Likert (assume interval)
  
  • how felt if not enough sleep
• Categorical
  
  • emotion is essential, useful, unnecessary
  • to behaviour of whether you get enough sleep

Question 17

What is an If-then condition?

1:39:35

Answer 17

• one of four complex multiple-choice format
• The test-taker must decide the consequence of one or more conditions being present

“If the true variance of a test increases but the error variance remains constant, which of the following will occur?”

a. Reliability will increase

b. Reliability will decrease
c. Observed variance will decrease
d. Neither reliability nor observed variance will change

if variance has an increased proportion of true variance - this will always increase reliability

Question 18

What is meant by the term multiple conditions in terms of item format?

1:41:05

Answer 18

• it is one of the four complex multiple-choice formats
• The test-taker uses two or more conditions in the statements listed to draw a conclusion

“Given that Mary’s raw score on a test is 60, the test mean is 59, and the standard deviation is 2, what is Mary’s z score?”

a. -2.00
b. - .50

c. .50

d. 2.00

z score = difference between raw score & mean / standard deviation

1 / 2 = .50

Question 19

What are multiple true-false items?

1:42:10

Answer 19

• one of four complex multiple-choice formats

The test-taker decides whether one, all, or none of the two or more conditions or statements listed in the stem are correct

“Is it true that (1) Alfred Binet was the father of intelligence testing and the (2) his first intelligence test was published in 1916?”

a. Both 1 and 2
b. 1 but not 2
c. 2 but not 1
d. Neither 1 nor 2

Question 20

What is an oddity item?

1:43:16

Answer 20

• one of four complex multiple-choice formats
• The test-taker indicates which option does not belong with the others
  • which is the odd one out?

“Which of the following names does not belong with the others?”

a. Alfred Adler

b. Sigmund Freud
c. Carl Jung
d. Carl Rogers

Freud, Jung & Rogers = listeners

Adler = confronter

Question 21

What is the first step in writing test items?

Answer 21

• create an item bank or pool
• start off with at least twice as many as you want to end up with

Question 22

What is a classification item format?

Answer 22

• one of four complex multiple-choice formats
• The test-taker classifies a person, object or condition into one of several categories designated in the stem

Jean Piaget is best described as a _____________ psychologist

a. Clinical

**b. Developmental **

c. Psychometric
d. Social