Psychometrics

Question 1

What factors influence evaluation of exam data?

Answer 1

Sample size and sample inclusion criteria (ex: first time, repeat)

Question 2

What is Item Response Theory (IRT)?

Answer 2

IRT models the likelihood of the test taker choosing the correct response given the test takers ability level and the item parameters. High Theta = high achievement. Generally ranges from -3 to +3

Question 3

What is reliability?

Answer 3

Consistency of scores across tests

Question 4

What is Standard Error of Measurement?

Answer 4

Standard deviation of individual’s observed scores from repeated tests - SEM is a function of standard deviation. For criterion referenced tests you need SEM at the cut.

Question 5

What is Decision Consistency Index?

Answer 5

The extent that pass/fail rate would be the same across repeated exams.

Question 6

What factors have the greatest influence on reliability?

Answer 6

of items (more items = more reliability), exam domain coverage, homogeneous test group (more similar = lower reliability). Reliability is a function of variance (low variance = low reliability).

Question 7

What is a p-value?

Answer 7

Proportion of test takers answering the item correctly