13 Validity and Reliability 2

Question 1

What is the classical test theory formula?

Answer 1

X = T + E 
Score = True Score plus Error

Question 2

What is the formula for the reliability index (expressed as r)?

Answer 2

r = true variance / actual variance

Question 3

If r (reliability index) is .9, what does this mean?

Answer 3

90% of observed score is the true score and 10% is error

Question 4

What are three issues with CTT in psychology?

Answer 4

1. It assumes the theoretical construct exists - i.e. if it doesn’t there may be nothing to measure
2. The equation assumes fluctuation is due to error but it may be due to changes in reality, e.g. learning effects
3. The error may not fluctuate because it is systematic (e.g. using a thermometer to measure velocity)

Question 5

What is the assumption of additivity in reliability estimates?

Answer 5

We assume elements we’re measuring can be added, assuming they are independent. But this is rarely true in personality – personality is rooted in biological functions, which interact. CCT doesn’t account for this.

Question 6

What is the assumption of independence of item scores?

Answer 6

After answering one question, bias can be introduced. This may include learning effects, guessing what is being tested etc.

Question 7

What is the assumption of equal dimensionality of scale items?

Answer 7

Assumption that all items can be Likertised in the same way

Question 8

Sources of measurement error among individuals may be __________ or ___________ ?

Answer 8

Sources of measurement error among individuals may be idiosyncratic or generic.

Question 9

Name four idiosyncratic sources of error when measuring individuals.

Answer 9

1. Language (cultural background)
2. Mood
3. Fatigue
4. Memory (endophenotypic error)

Question 10

Name four generic sources of error when measuring individuals.

Answer 10

1. Faking (social desirability - impression formation; self-deception)
2. Acquiescence and nay-saying bias
3. Midpoint extreme response styles (floor/ceiling effects)
4. Random responses

Question 11

Sources of measurement error within items may be ________, __________ or ___________

Answer 11

Sources of measurement error within items may be content-related, format-related or administration related

Question 12

Name two content-related sources of error in measurement items.

Answer 12

1. Lack of clarity

2. Leading/biased questions

Question 13

Name two format-related sources of error in measurement items.

Answer 13

1. Range and biases in content domain coverage

2. Number of response categories (too many/few)

Question 14

Name three administration-related sources of error in measurement items.

Answer 14

1. Learning/training (of the participants)
2. Distracting settings
3. Administration variations

Question 15

What is internal consistency reliability?

Answer 15

The degree of consistency (homogeneity) in responses to scale items that measure the same construct.

Question 16

What does Cronbach’s alpha measure?

Answer 16

Internal consistency reliability –how much different items measuring the same construct correlate with each other.

Question 17

What does a Cronbach’s alpha (r alpha) of 0 mean?

Answer 17

Only error variance is present in the measurement

Question 18

What does a Cronbach’s alpha (r alpha) of 1 mean?

Answer 18

Only true scores are present in the measurement

Question 19

What is the usual cut-off range for Cronbach’s alpha?

Answer 19

Between .6 and .8

Question 20

How can you increase the value of Cronbach’s alpha?

Answer 20

By inserting more relevant (homogenous) items in the measurement.

Question 21

What is split-half reliability?

Answer 21

The estimated reliability based on the correlation of two equal parts of a measurement.

Question 22

What method is used to calculate split-half reliability?

Answer 22

The Spearman-Brown split-half reliability coefficient

Question 23

What is test-retest (temporal) reliability?

Answer 23

The stability of test scores over time. When the same test is administered to same people at least two different times.

Question 24

In what two ways is test-retest reliability measured?

Answer 24

1. Pearson correlation - tests concurrent variation (rank-order stability)
2. t-test –tests mean drift (mean level stability)

Question 25

What are three issues in ascertaining test-retest reliability?

Answer 25

1. Dropouts / non-response rates (bias)
2. Temporal instability of constructs, e.g. anger
3. Optimal time interval

Question 26

What is parallel forms reliability?

Answer 26

Similar to split-half, but estimation is based on two equivalent forms of a measurement (not necessarily two halves of same measurement)

Question 27

What is inter-scorer reliability?

Answer 27

When estimated reliability is based on the correlation between two or more judges’ ratings of an item or scale. I.e. several judges score the same test.

Question 28

What is the standard error of measurement (SEM)?

Answer 28

An index of the average degree of random error in an individual’s observed score. Basically, it’s an indication of the dispersion of the measurement errors when you are trying to estimate participants’ true scores from their observed test scores.

Question 29

What is the formula for standard error of measurement (SEM)?

Answer 29

SEM = variance of test scores multiplied by the square root of 1 – r (Cronbach’s alpha)

Question 30

Why is SEM used?

Answer 30

It helps researchers identify a range of observed scores (confidence interval) within which the true score of a measurement or an individual is expected to lie

Question 31

What percentage of scores fall within 1, 2, 3 SEMs of the true score?

Answer 31

68% within 1
95% within 2
99% within 3

Question 32

What is the critical SEM for a confidence interval of 95%?

Answer 32

+/-1.96 SEM