Psychometrics: reliability

Question 1

What is a reliable test?

Answer 1

• consistency in measurement

- the precision with which the test score measures achievement

Question 2

What is reliability

Answer 2

• the desired consistency or reproductibility of test scores (does it give me the same accurate measurement each time it is used?)
• no test is free from error

Question 3

Reliability formula

Answer 3

x=T+e

x- the observed score
T- the true score
e- the error

Question 4

The Four Assumptions of Classical Test Score Theory

Answer 4

1. Each person has a true score we could obtain if there was no measurement error
2. there is measurement error- but this error is random
3. the true score of an individual doesnt change with repeated applications of the same test, even though their observed score does
4. the distribution of random errors and thus observed test scores with be the same for all people

Question 5

Standard Error of measurement (SEM)

Answer 5

-works out how much measurement error we have by working out how much on average, an observed score on our test differs from the true score
(standard deviation)

Question 6

Problems with Classical Test Score Theory

Answer 6

1. Population dependent
2. Test dependent
3. Assumption of equal error measurement

Question 7

Domain Sampling Model

Answer 7

• a central concept of Classical Test Theory
• cant ask all possible questions on a test so only use a few test items (sample)
• using fewer test items can lead to the intro of error
• as sample gets larger, estimate is more accurate

Question 8

4 Types of reliability

Answer 8

1. Test-retest reliability
2. Parallel forms reliability
3. Internal consistency
4. inter-rater reliability

Question 9

Test-retest reliability

Answer 9

• give someone a test and then give them the same test later on
• if scores are highly correlated, we have a good test-retest reliability
• correlation between 2 scores = co-efficient of stability
• time sampling

Question 10

Issues with test-retest

Answer 10

• . can it be used when measuring mood/stress?
• scores increase because done them before
• if thing being measure changes?
• what if an event happens between tests administrations to change the thing being tested?

Question 11

Parallel forms reliability

Answer 11

• 2 forms of the same test (questionnaire with different items)
• correlation between the two = co-efficient of equivalence
• item sampling

Question 12

Ways to change test in parallel forms reliability

Answer 12

• question response alternatives are reworded
• order is changed
• change wording of question

Question 13

Issues with parallel forms reliability

Answer 13

• what if different forms are given at two different times?
• do you give the form to the same or different people?
• what if people work out how to answer the one form from doing the other form?
• do you have two forms of the test and/or do we want to develop two forms of the same test?

Question 14

Internal Consistency

Answer 14

-do different items within a test all measure the same thing, to an extent?

Question 15

Examples of internal consistency tests

Answer 15

• split-half reliability
• KR20
• coefficient alpha

Question 16

Split-half reliability

Answer 16

• test split in half and each half scores separately

- total scores for each half are correlated

Question 17

advantage of split-half reliability

Answer 17

-only need one test (dont need 2 forms)

Question 18

challenge of split-half reliability

Answer 18

-how to divide the test into equivalent halves

Question 19

issues with split-half reliability

Answer 19

• by splitting test, have less items and the lower the reliability will be
• correlation changes each time depending how items are split

Question 20

Spearman-Brown formula

Answer 20

is the solution to the problem for split tests- that each half will have reduced reliability compared to the total test)

Question 21

Coefficient/Cronbach’s Alpha

Answer 21

• estimates the consistency of responses to different scale items
• takes the average of all possible split-half correlations for a test

Question 22

What do coefficients results mean? Cronbach’s A

Answer 22

0- no consistency in measurement

1- perfect consistency in measurement

Question 23

what level of reliability is appropriate? Cronbach’s A

Answer 23

0. 7 - exploratory research
1. 8 - basic research
2. 9 - applied scenarios

Question 24

Cronbach’s alpha can be affected by

Answer 24

1. multidimensionality
2. bad test items
3. number of items

Question 25

Inter-rater reliability

Answer 25

• measures how consistently 2 or more judges agree on rating something
• by correlating raters scores

Question 26

Cohen’s kappa

Answer 26

-2 judges/raters
-ranges from 1 (perfect agreement) to -1 (agreement less than would be expected by chance)
>0.75 - excellent agreement
0.4-0.7 - satisfactory

Question 27

Fleiss’ kappa

Answer 27

for 2 or more judges/raters

Question 28

Intra-class correlation (ICC)

Answer 28

used for inter-rater reliability when rating interval and ordinal measurements

Question 29

ICC vs COhen/Fleiss kappa

Answer 29

• ICC for continuous data (interval and ordinal)

- kappa for observations in a category (nominal/categorical data)

Question 30

SEM calculation

Answer 30

SEM= S(sqrt 1-r)
s- stdev
r- reliability of test

Question 31

confidence intervals using SEM

Answer 31

• z score for 95% confidence interval= 1.96
• lower bound = x- 1.96*SEM
• upper bound =x+ 1.96*SEM

Question 32

Factors influencing reliability

Answer 32

1. number of items in scale
2. variability of the sample (better with wider population)
3. extraneous variables (testing situation, ambiguous items, unstandardised procedures, perceived demand effect)

Question 33

how to improve reliability

Answer 33

2. item analysis
3. Use identical instructions
4. Eliminate questions that evoke inconsistent responses
5. Cover entire range of the dimension
6. Clear conceptualization
7. Standardization
8. Inter-rater training
9. Use more precise measurement
10. Use multiple indicators
11. Pilot-testing