Psychometrics - Reliability + Coefficient Alpha

Question 1

What is reliability?

Answer 1

The desired consistency or reproducibility of test scores

Question 2

4 assumptions of classical test theory

Answer 2

1. Each person has a true score that we could obtain if it weren’t for measurement error
2. There is measurement error, but it’s random
3. The true score of a person doesn’t change upon repeating tests, even though the observed score does
4. The distribution of random errors will be the same for all ages

Question 3

Domain Sampling Model

Answer 3

The idea that we can’t construct a text that asks all possible questions within the domain being tested, so we have to select only certain ones. But using fewer items can lead to an introduction of error

Question 4

Reliability analysis’s aim:

Answer 4

Establish how much error is made by using the score from the shorter test as an estimate of one’s true ability - error comes from multiple sources, and there are different ways of measuring reliability that are sensitive to different measurement errors

Question 5

Four types of reliability

Answer 5

1. Test-retest
2. Parallel forms
3. Internal consistency
4. Inter-rater reliability

Question 6

What kind of error is the test-retest method designed for?

Answer 6

Time sampling

Question 7

How does test-retest work?

Answer 7

You give someone the same test at two different points in time, and assess how much of difference there is in performance from the first test taking to the second

Question 8

Problems with test-retest reliability

Answer 8

Practice effect, testing effects, maturation, history

Also not ideal when you want to assess something that is expected to change over time

Question 9

What source of error is parallel sampling designed to account for?

Answer 9

Item sampling

Question 10

How does parallel forms reliability work?

Answer 10

You compose two different forms of the same test and get participants to do both

Question 11

Problems with parallel forms of reliability

Answer 11

How do we give both tests without having time problems? You need a bigger item pool. Testing effects.

Question 12

What error does internal consistency reliability account for?

Answer 12

The reliability of one test administered on one occasion

Question 13

What does internal consistency measure and what three methods are used?

Answer 13

Do the different items within all measure the same thing to the same extent?

1. Split half reliability
2. Coefficient alpha
3. KR-20

Question 14

When do we use KR-20?

Answer 14

It’s used to find the alpha with dichotomous format measures

Question 15

How does split half reliability work?

Answer 15

A test is split in half, assessed and then correlated to see if the test is consistent.

Question 16

Pros and cons of split half

Answer 16

+ You only need one test

• It’s difficult to split the test
• Halving the length of the test means it’s less reliable
• Split half will change each time depending on which items sit in which test

Question 17

How do we account for the decrease in reliability that splitting the test will have?

Answer 17

Spearman-Brown correction!

Question 18

Formula for Spearman-Brown

Answer 18

Predicted reliability = 2 x correlation between halves / 1 + correlation between halves

Question 19

How do we account for the differences in Split Half depending on which items are where?

Answer 19

Cronbach’s Alpha!

Question 20

What does Cronbach’s alpha do?

Answer 20

It takes the average of all possible split half correlations for a test

Question 21

Formula for Cronbach’s alpha

Answer 21

A = (number of indicators)(average inter-item correlation)/(1 + (number of indicators - 1)(average inter-item correlation)

= kr/(1 + (k - 1)r)

Question 22

Alpha: correlation between number of items and reliability

Answer 22

Positive and non-linear, there is a rapid increase from 2-10, steady from 11-39 and it plateaus around 40.

Question 23

What levels of Cronbach’s alpha are we aiming for?

Answer 23

0. 7 for exploratory research
1. 8 for basic research
2. 9 for applied scenarios

Question 24

What affects Cronbach’s alpha?

Answer 24

• Multidimensionality
• Bad test items
• Number of items

Question 25

What source of error does inter-rater reliability help with?

Answer 25

Observer differences

Question 26

What does inter-rater reliability do?

Answer 26

It measures how consistently two or more rafters agree on rating something, based on the premise that multiple raters fro measurement can improve measurement reliability

Question 27

What inter-rater reliability do we use when?

Answer 27

2 raters = Cohen’s kappa

3+ = Fleiss’ kappa

Question 28

When do we use Cronbach’s alpha?

Answer 28

When we’re using raw scores

Question 29

When do we use standardized item alpha?

Answer 29

When scores have been standardized to account for things like age, gender etc.

Question 30

What is inter-item variance?

Answer 30

The variance between two items