lecture 2- reliability

Question 1

reliability is a … property of a test

Answer 1

fundamental

Question 2

if a test is …. it will never be valid ie reliability is a necessary (but obviously not sufficient) condition for validity

Answer 2

not reliable

Question 3

who is reliability important for?

Answer 3

particularly important for applied psychologists (clinical psychologists, clinical neuropsychologists, educational psychologist) as they deal with individual areas

Question 4

what do reliability coefficients tell us?

Answer 4

tell us how much of the variability in scores on tests is true variability (ie signal) and how much of it is measured error (ie noise)

Question 5

if a psychological test has a reliability coefficient of (say) 0.8, then 80% of the variability in scores is true variability (ie the test is picking up real differences in the construct being measured)
- it follows that 20% of the variability in scores reflects measurement error ie noise in the instrument

Question 6

what can the reliability coefficient be seen as?

Answer 6

the reliability coefficient can be seen as a signal-to-(signal plus noise) ratio

Question 7

what is the formula for reliability?

Answer 7

reliability(ie r11) = true variance/ total variance

Question 8

what does r11 mean?

Answer 8

because it can be seen as the tests correlation with (a strictly parallel version of ) itself - there is always measurement error so the correlation is not perfect

Question 9

what does reliability allow us to do?

Answer 9

allows us to quantify the confidence we have in our test results and allows us to assess whether differences between an individuals scores are liable to reflect true differences in ability or may have simple arisen by chance

Question 10

what are psychologists warned not to do?

Answer 10

are often warned not to reify a test score: it is only an estimate of an individuals true ability level or mood level ect

Question 11

what do reliability coefficients allow us to form?

Answer 11

allow us to form confidence intervals on scores to help remind us of the above

Question 12

what can lead psychologist astray?

Answer 12

as much of clinical practice is concerned with differences between an individuals abilities, a failure to consider the reliability of measures can lead the psychologist astray

Question 13

chapman and chapman (1993) provided a classic illustration of artefacts arising from differences in reliability

Answer 13

• skits patients were compared to a healthy control sample on two tasks
• the skits sample appeared to have a severe deficit on only one of the tasks
• was in fact the same task but one version rendered less reliable (by shortening the task)

Question 14

reliability coefficients, how high should they be?

Answer 14

there is no absolute rule (will depend on purpose) but various standards have been proposed:
- Nunnally and Bernstein (1994) take a hard line and propose that reliability coefficients should be above 0.90

others are less demanding:
- sattler (2001) suggests that tests with reliabilities of 0.70 and above should be considered to be “reliable”
- similarly, Cicchetti (1994) suggests tests with reliabilities below 0.70 should be considered “unreliable”

Question 14

can reliability be too high?

Answer 14

yes, if we are trying to measure a broad, multifaceted, construct then a very high reliability may indicate a problem (Boyle, 1985)

Question 15

state- if a test is not reliable it will never be valid ie reliability is a necessary (but obviously not sufficient) condition for validity

Question 16

state- to be considered reliable a test should provide a consistent measure

Question 17

how can we decide if a test is reliable?

Answer 17

• cronbachs alpha
• test-retest reliability

Question 18

what is the standard way of calculating the reliability of tests?

Answer 18

the standard way of calculating the reliability of tests is to use Cronbachs alpha

Question 19

what is cronbachs alpha is determined by?

Answer 19

-the number of items in the test
- the size of the correlations between the items

-longer tests are more reliable
- tests in which the items have higher correlations with each other are more reliable

Question 20

reliability and test length: a Caveat

Answer 20

• longer texts will be more reliable only provided other things are equal
• if the reliability is disappointing, simply throwing in a bunch of additional poor items (items that are not closely related to the other items or have ceiling or floor effects) will not help much
  -so we should really say, longer tests are more reliable provided that the items in the longer test are as good (as highly correlated with the other items) as the shorter version

Question 21

psychologists are always looking for ways to save time and try and develop short-forms of tests..

Answer 21

• sometimes this can be done with only a marginal lowering of reliability because poor items (ig items that are not highly correlated with the other items) are selectively dropped

Question 22

reliability coefficients for WAIS-IV: subsets

Answer 22

• for use with individuals we would want reliability to be at least above 0.8
• in general reliabilities of the subtests are impressive

the processing speed subsets are a bit lower than others but
- these were assessed using test-retest method and
- are still acceptable in any case

reliability coefficients were fairly consistent across age bands

Question 23

state- the reliability of a composite (an index or IQ in this case) is a function of the reliability of components (subsets) and the correlation between the components

Question 24

what will composites always have?

Answer 24

composites will always have superior reliability to the components they are derived from if the components are correlated (and they always are)
- can see this when compare the reliabilities of the WAIS-IV subsets with those for the indexes

Question 25

reliability coeffcients for WAIS - IV IQs and indexes

Answer 25

• the reliability of WAIS-IV indexes and FSIQ are uniformly excellent - among the highest of any psychological instrument
• in case of FSIQ (in both US and UK), r11 is 0.98 so..
  -98% of the variance in test scores is true variance and only 2% is measurement error
• reliability for processing speed is bit lower than others- in part because it is a composite made up of only two components (coding and symbol search)

Question 26

what does temporal stability refer to?

Answer 26

• temporal stability refers to the extent to which a measure yields consistent scores over time, ie stability coefficients allow us to gauge extent to which performance is affected by day to day fluctuations/ differences in mood, testing conditions ect

Question 27

what is temporal stability assessed by?

Answer 27

assessed using the test-retest method

Question 28

state- the temporal stability or test-retest reliability of a scale is simply the correlation between scores at test and retest

Question 29

temporal stability contd

Answer 29

-normally the interval between administrations is set so it is unlikely that true change has occurred in the underlying ability
- against that must be set the need to avoid inflating the estimate of stability due to a testees memory for their previous answers

Question 30

stability coefficients for WAIS-IV IQ and indexes

Answer 30

• just as was case for reliability (internal consistency), temporal stability of a composite will be higher than for its components - some of chance fluctuations on components will cancel each other out
• temporal stability of FSIQ and indexes are highly satisfactory , again among the highest of any psychological instrument
• temporal stability of mental tests is generally very impressive

Question 31

practice effects on cognitive tests

Answer 31

• a psychologist often wants to know if an individuals cognitive abilities have genuinely improves
• similarly, a psychologist often wants to know if an individuals cognitive abilities have genuinely deteriorated
• a complication is that there are practice effects on most cognitive tests

Question 32

what are the consequences of practice effects?

Answer 32

• may exaggerate or give false impression of recovery/improvement
• may mask a deterioration in functioning