Test development

Question 1

What are two types of validity used when developing a test?

Answer 1

Substantive validity and Structural validity.

Question 2

What is the aim of substantive validity?

Answer 2

We are aiming to form a clear idea of the construct we want to measure, both what it is, and what it is not.

Question 3

What are the two steps of substantive validity?

Answer 3

Conceptualisation

Create an item pool

Question 4

What are the three main selection strategies used in structural validity?

Answer 4

Empirical/criterion-based
Internal consistency (factor analysis)
Item response theory

Question 5

What type of validity is substantive validity trying to address?

Answer 5

Content validity

Question 6

What are the two dominant formats used in substantive validity?

Answer 6

Dichotomous responses

Likert-type rating scales

Question 7

What is the aim of empirical/criterion-based item selection strategies?

Answer 7

To create tests that discriminate between two groups (high and low scores).

Question 8

How is internal consistency tested?

Answer 8

Remove one item at a time and retest Cronbach’s alpha to see if we have a higher coefficient. If the coefficient rises significantly we leave the item out of the test.

Question 9

What does Factor analysis achieve?

Answer 9

It allows us to group different variables together that make up a construct.

Question 10

What are three ways we test for internal consistency?

Answer 10

Cronbach’s alpha
Item-total score correlation
Factor analysis

Question 11

How does content, criterion and construct validity differ from substantive and structural validity?

Answer 11

Content, criterion and construct validity are more a measure of external validity (how scores relate to phenomenon outside of the test itself). While Substantive and structural are internal measures of validity.

Question 12

What are the three features of an item characteristic curve?

Answer 12

Discrimination (steepness of the slope)
Difficulty (Position of the curve)
Pseudoguessing (intersection with Y-axis)

Question 13

What is ideal in regards to pseudo guessing?

Answer 13

We want our ICC to cross the y-axis at 0

Question 14

What does a steeper slope on an ICC tell us?

Answer 14

There is increased discrimination in the test.

Question 15

What does the position of the ICC tell us?

Answer 15

Further right indicates the test is harder

Question 16

What sort of measurement problems can item response theory be used to solve?

Answer 16

Test bias ( differential item functioning)

Question 17

What is meant by test bias and differential item functioning?

Answer 17

This is when two groups of equal ability have different probabilities of endorsing an item on a test. For example, men and women answer a question on anxiety differently, therefore do we want the item in the test as it might not unbiasedly represent anxiety.

Question 18

What is the difference between Classic test theory (CTT) and Item Response theory (IRT)?

Answer 18

Classic test theory is a more rigid form of testing that uses dichotomous or Likert-type scales for testing. While Item response theory is a more adaptive form of testing that can adapt to a participant’s answers. ITR can be more costly.