reliability

Question 1

what is reliability a measure of

Answer 1

consistency

Question 2

3 ways of assessing reliability

Answer 2

1. test-retest
2. inter-observer reliability
3. measuring reliability by correlational analysis

Question 3

how can test-retest be used to assess reliability

Answer 3

= administering same test/questionairre to same person (or people) on different occasions

• if reliable, the results obtained should be the same (or very similar) each time they’re administered
• most commonly used with questionnaires & psychological tests (eg. IQ) but also applied to interviews
• must be sufficient time between test & retest to ensure the participant/respondent cannot recall their previous answers, but not so long that their attitudes, opinions or abilities have changed
• if questionnaire/test, the answers from these 2 tests would be correlated to ensure they’re similar & if significant (& positive), the reliability of the measuring instrument is good

Question 4

how can inter-observer reliability be used to assess reliability

Answer 4

= conduct observations in teams of at least 2

• one observers interpretation of events may differ from someone else’s, causing subjectivity, bias & unreliability to data collected
• may involve small-scale trial run (pilot study) of observation to check observers are applying behavioural categories in same way, or comparison reported at end of study
• observers watch same event but record data independently
• data collected by each observer correlated to assess reliability

Question 5

how is correlational analysis used measure reliability

Answer 5

• in test-retest & inter-observer reliability, two sets of scores are correlated
• if correlation coefficient exceeds +.80 = reliable

Question 6

improving reliability using questionnaires

Answer 6

• reliability of questionnaires tested by test-retest
• two data sets compared & should produce correlation coefficient exceeding +.80
• if questionnaire produces low test-retest reliability, some of items may need to be ‘deselected’ or rewritten
• eg. replace some open questions (room for misinterpretation) with closed, fixed-choice alternatives which are less ambiguous

Question 7

improving reliability using interviews

Answer 7

• ensure reliability by using same interviewer each time
• if unable to/impractical, all interviewers must be trained properly (eg. so one interview isn’t asking too leading or ambiguous questions)
• more easily avoided in structured interviews as behaviour of interviewers controlled by fixed questions = more reliable

Question 8

improving reliability using observations

Answer 8

• ensure behavioural categories are fully operationalised, so they’re measureable and obvious
• categories shouldn’t overlap & all possible behaviours must be covered on checklist
• if categories aren’t operationalised well, or overlapping/absent, different observers will make own judgements of what to record & may end up with different/inconsistent records
• if low reliability, observers may need training in use of behavioural categories &/or wish to discuss decisions with each other so categories applied more consistently

Question 9

improving reliability using experiments

Answer 9

• to compare performance of different participants, prodedures must be the same every time
• this means an experimenter is concerned with standardised procedures