Reliability and coefficient alpha

Question 1

What is reliability?

Answer 1

The desired consistency or reproductibility of test scores

-does my test give me the same accurate measurement each time?

Question 2

Test score theory

Answer 2

every person has a true score that we can measure but no test is free from error

x=T+e (x=observed score, T= true score, e= error)

Question 3

Classical test theory: 4 assumptions

Answer 3

1. each person has a true score we could obtain if there was no measurement error
2. there is measurement error but this error is random
3. the true score of an individual doesn’t change with repeated applications of the same test even though their observed score does
4. the distribution of random errors will be the same for all ages

Question 4

Classical test theory:

The domain sampling model

Answer 4

If we construct a test on something, we can’t ask all possible questions

• > So we only use a few test items (sample)
• Using fewer test items can lead to the introduction of error

Question 5

The domain sampling model

formula

Answer 5

reliability = variance of observed scores on short test/variance of true scores

As the sample gets larger, estimate is more accurate

Question 6

Other things can affect performance…

Answer 6

• might be tired on day taking the test (different scores for different days)

Question 7

Types of reliability

Answer 7

1. test-retest reliability
2. parallel forms reliability
3. internal consistency reliability (split half, Kuder-Richards 20, Cronbach’s alpha)
4. inter-rater reliability

Question 8

Test-retest reliability

Answer 8

• Give someone the same test at two different points in time.
• If the scores are highly correlated, we have good test-retest reliability
• Correlation between the 2 scores also known as the co-efficient of stability

Question 9

Source of error in test-retest reliability

Answer 9

time sampling

Question 10

Issues with test-retest reliability

Answer 10

Can we use it when measuring things like mood, stress, etc.?

Won’t the person’s score increase the 2nd time because of practice effect?

What if we want to measure changes between 1st and 2nd administration?

Can the actual experience of being tested change the thing being tested?

What if some event happens in between the 1st and 2nd administration to change the thing being tested?

Question 11

Parallel forms reliability

Answer 11

• Two different forms of the same test (i.e., measuring the same construct)
• Correlation between the two forms known as the co-efficient of equivalence

Question 12

Parallel forms reliability- source of error

Answer 12

item sampling

Question 13

Parallel forms reliability- Ways to change the form of test

Answer 13

• question response alternatives are reworded
• order is changed (reduce practice effect)
• change wording of question

Question 14

Parallel forms reliability: issues

Answer 14

What if we give the different forms to people at two different times?

Do we give the different forms to the same people, or different people?

What if people work out how to answer the one form from doing the other form?

Difficult to generate a big enough item pool

Question 15

Internal consistency reliability

Answer 15

Do the different items within one test all measure the same thing to the same extent?
I.e., Are items within a single test highly correlated?
Split-half reliability
Coefficient alpha

Question 16

Internal consistency reliability: source of error

Answer 16

-internal consistency/reliability of on test administered on one occasion

Question 17

Split-half reliability

Answer 17

A test is split in half
Each half scored separately
Total scores for each half correlated

Question 18

Split-half reliability- advantage

Answer 18

• we only need 1 test- don’t need 2 forms

Question 19

Split-half reliability- disadvantage

Answer 19

-challenging to divide the test into equal halves

Question 20

SPEARMAN-BROWN CORRECTION

Answer 20

solves the problem of split half tests having reduced reliability compared to the total test

rsb= 2r(hh)/1+r(hh)

r(sb)- predicted reliability
r(hh)- reliability of the current test(correlation between halves)

Question 21

Split-half reliability: issues

Answer 21

1. We have taken one test and split it into two tests that are half the length – won’t this underestimate reliability?

Example: We have a test of 20 items, split in half, and correlate each half
Similar to 2 tests of 10 items
The fewer items we have, the lower our reliability

2. Won’t the correlation change each time depending on which items we put in each half?

Yes – we will get a different reliability coefficient for each different split

Ideally the halves should be equivalent

Question 22

Coefficient/cronbach’s alpha

Answer 22

Takes the average of all possible split-half correlations for a test

a=kr/(1+(k-1)r)

k- number of indicators
r- mean inter-rater correlation

Question 23

cronbach’s alpha

-number of items

Answer 23

Rapid increase in internal consistency reliability from 2 to 10 items
Steady increase from 11 to 30
Tapers off after about 40 items

Question 24

Interpreting cronbach’s alpha

Answer 24

0. 00 = no consistency in measurement
1. 00 = perfect consistency in measurement
2. 70 = exploratory research
3. 80 = basic research
4. 90 = applied scenarios

Question 25

Cronbach’s alpha can be affected by:

Answer 25

Multidimensionality
Bad test items
Number of items

Question 26

inter-rater reliability

Answer 26

Measures how consistently 2 or more raters/judges agree on rating something

Question 27

Cohen’s kappa &Fleiss’ kappa

Answer 27

Cohen-2 raters/judges

Fleiss- more than 2 raters/judge

Ranges from 1 (perfect agreement) to -1
>.75 excellent agreement
.50-.75 satisfactory

Question 28

source of error of split half, alpha and Kr-20?

Answer 28

internal consistency

Question 29

source of error of Kappa?

Answer 29

observer differences

Question 30

What is coefficient alpha?

Answer 30

-Coefficient (Cronbach’s) alpha is one way of calculating the reliability of a test
-Specifically, it tells us about the internal consistency of a test
-Coefficient alpha measures the error associated with each individual test item
As well as error associated with how well the test items fit together

Question 31

What is coefficient alpha? 
 according to (cortina, 1993)

Answer 31

The mean of all split-half reliabilities

A measure of first-factor saturation

Question 32

Cronbach’s alpha vs standardised item alpha

Answer 32

Cronbach’s alpha
Deals with variance and covariance (the amount by which variables/items vary together, i.e., co-vary)

Standardized item alpha
Deals with the interim correlations (i.e., the correlation of each item with every other item)and the sum of the correlation matrix of each item with each other

Question 33

When do we use Cronbach’s alpha?

Answer 33

• used when you want to use raw scores (the actual score)

- item variance affects the score and CA takes this into account

Question 34

when do we use Standardised item alpha?

Answer 34

Standard scores are scores that have been transformed by taking into account things like age, in particular
E.g., raw scores on any WAIS subtest are converted to standard scores with a mean of 10 and a SD of 3

Question 35

Cronbach’s alpha is calculated by…

Answer 35

using inter-item variance and co-variance

Question 36

Variance

Answer 36

a measure of how scores vary

• usually taken as a measure of error
• Because if scores differ a lot between people, the item or test is not very accurate – less accuracy means more error

Question 37

Covariance

Answer 37

a measure of how much scores on items go together
E.g., if a person with a high score on Item 1 of a test also gets a high score on Item 2 of a test
Then items 1 and 2 of the test have a lot of co-variance (shared variance)