16 Putting it all together

Question 1

What’s the difference between a valid measure and a reliable measure (target analogy)?

Answer 1

A valid measure always hits the target; a reliable measure always hits the same spot on the target.

Question 2

Reliability is a ________ but not ________ condition for validity.

Answer 2

Reliability is a NECESSARY but not SUFFICIENT condition for validity.

Question 3

At the conceptual level, a _____ measure is always reliable

Answer 3

At the conceptual level, a valid measure is always reliable.

Question 4

Fuzzy reliability/validity distinction. Cronbach’s alphas can be an index of ______ ________ ________ or _________ ________ _______

Answer 4

Cronbach’s alphas can be an index internal consistency reliability or internal consistency validity (related to construct validity)

Question 5

Fuzzy reliability/validity distinction. Parallel-forms reliability also assesses what kinds of validity?

Answer 5

Concurrent validity and to a degree convergence validity.

Question 6

Assessment of inter-rater reliability is closely related to what form of validity?

Answer 6

Content validity

Question 7

Test-retest reliability can be used an index of __________ validity

Answer 7

Test-retest reliability can be used an index of external validity

Question 8

What happens to standard error of measurement when you increase sample size?

Answer 8

It gets smaller. SEM is inversely proportionate to the square root of sample size.

Question 9

What’s the difference between Q (respondent-centred) and R (stimulus-centred) analysis?

Answer 9

Q (respondent-centred) analysis examines systematic variation across respondents –i.e. how responses differ between people.

R (stimulus-centred) analysis focuses on how responses differ between stimuli (test items).

Question 10

What could be the problems caused by non-discriminating items?

Answer 10

Invariable responses, ceiling/floor effect.

Question 11

What are serial effects?

Answer 11

The effect of previously answered questions on subsequent questions

Question 12

What are homogenous and inversely keyed items?

Answer 12

Homogenous: 
"I go to parties" 
"I go to social gatherings"
Inversely keyed
"I tend to go to parties"
"I avoid parties"

Question 13

Why do homogenous and inversely keyed items useful?

Answer 13

Because they detect invariable or random responses.

Question 14

What is generalisability theory?

Answer 14

A statistical framework developed by Cronbach for determining how well scores can be generalised to another setting. It attempts to understand all the sources of variation –called facets –(time, persons, raters, setting) and predict scores based on the variation of these facets.

Question 15

What’s the main difference between generalisability theory and classical test theory?

Answer 15

Classical test theory has just one error term:
X = T + E
Generalisability theory allows for various sources of error which may vary independently.

Question 16

What is Item Response Theory?

Answer 16

Item Response Theory is the study of test and item scores based on assumptions concerning the mathematical relationship between abilities (or other hypothesized traits) and item responses. It models the relationship between a respondent’s ability and the probability of getting a correct response on an item. I.e. smart – more likely to be correct. It can then be used to predict scores from one test to another, and for other purposes.