Reliability

Question 1

what is reliability?

Answer 1

a measure of consistency

• if a measurement is made twice and produces the same result - the measure is reliable

Question 2

what are the ways of assessing reliability?

Answer 2

test re-test

inter-observer reliability

measuring reliability

Question 3

what is test re-test?

Answer 3

it involves administering the same test/ questionnaire to the same person/ people of a different occasion

if reliable, the results should be the same or very similar each time

(this can be used with questionnaires, psychological tests and interviews)

• there must be sufficient time between test and retest to ensure the participant can’t recall their answers but not long that they change opinion

Question 4

how does test re-test work with questionnaires?

Answer 4

the two sets of scores would be correlated to make sure they are very similar

if the correlation turns out to be significant (and positive) then the reliability of the measuring instrument is assumed to be good

Question 5

what is inter-rater reliability?

Answer 5

one person’s observation may interpret event different to someone else - introducing subjectivity bias and unreliability into the data collection process

observers should be in teams of at least two

inter observer reliability must be established - could be done using a small scale pilot study to ensure observers are applying the behavioural categories in the same way

data collected by the two observers should be correlated to assess reliability

Question 6

what is measuring reliability?

Answer 6

reliability is measured using correlational analysis

in test -re-test, and inter -observer reliability, the 2 sets of scores are correlated

the correlation coefficient should exceed +80 for reliability

Question 7

how can reliability be improved with questionnaires?

Answer 7

using test re-test method - comparing 2 sets of data should produce a correlation that exceeds +80

• questionnaire producing low-test reliability may require some of the items to be ‘deselected’ or rewritten
• some open ended questions may be changed with fixed choice alternatives that may be less ambiguous

Question 8

how can reliability be improved in interviews?

Answer 8

using the same interviewer each time

or to train the interviewers so that questions are not leading or ambiguous

using structured interviews where the interviewer’s behaviour is more controlled by the fixed questions

• unstructured interviews are less likely to reliable

Question 9

observations improving relaibility

Answer 9

make sure behavioural categorised are properly operationalised and they are measurable and self evident

categories should not overlap and all behaviours should be covered

if categories are not operationalised well or overlapping or absent, different observers may make their own judgements - leading to inconsistent records

observers may need training on the use of behavioural categories

Question 10

improving reliability of experiments

Answer 10

the procedures need to be the same every time - they should be standardised procedures.