reliability Flashcards
what is reliability?
a measure of how consistent the findings from an investigation are
what is reliability important?
● To ensure the DV is being measured accurately.
● To ensure that over periods of time, the outcome is still the same.
● To ensure that all the conclusions made are accurate and valid as if not can have implications for theory development.
types of reliability: internal reliability
describes how consistent something is within itself.
types of reliability:external reliability
this is when consistent results are produced regardless of when the investigation is used or who administers it.
what is test-retest reliability?
A method of assessing the reliability of a questionnaire or psychological test by assessing the same person on two separate occasions. This shows to what extent the test (or other measure) produces the same answers ie. is consistent or reliable
what is inter-observer reliability?
The extent to which there is agreement between two or more observers involved in observations of a behaviour. This is measured by correlating the observations of two or more observers.
A general rule is that if (total number of agreements) / (total number of observations) > + 80, the data has high inter-observer reliability.
test-retest
This method checks if a test or questionnaire gives consistent results over time.
The same test is given to the same group of people on different occasions.
If the results are similar each time, the test is considered reliable.
A correlation test is used to compare the two sets of scores, and if they correlate positively and significantly, the tool is deemed reliable.
This is often used with questionnaires and psychological tests but can also be applied to interviews
inter-observer reliability
This addresses the issue of subjectivity in observational research, where different observers may interpret things differently.
It’s crucial that multiple observers’ interpretations are consistent.
A pilot study (trial run) may be conducted to check this reliability.
Inter-observer reliability ensures that different observers have similar judgments.
measuring reliability
Reliability is measured using correlational analysis.
In both test–retest and inter-observer reliability methods, the correlation between two sets of data should be at least +0.80 for the data to be considered reliable.
The goal of these methods is to ensure that tests and observations provide consistent and accurate results over time or across different observers
ways of improving reliability
- questionaires
- interviews
- observations
- experiments
improving reliability: questionnaires
A questionnaire that produces low test-retest reliability may require some of the items to be ‘deselected’ or rewritten. One solution might be to replace some of the open questions where there may be more room for misinterpretation, with closed, fixed choice alternatives which may be less ambiguous.
improving reliability: interviews
For interviews, probably the best way of ensuring reliability is to use the same interviewer each time. If this is not practical or possible, all interviewers must be properly trained, so for example, the interviewer should not ask leading or ambiguous questions. They should all be able to structure their interviews in a ‘certain manner’ which can be followed by all to ensure that everything is similar. This is more easily avoided in a structured interview where the interviewer’s behaviour is more controlled by the fixed questions. Interviews that are unstructured and more ‘free-flowing’ are less likely to be reliable.
improving reliability: observations
The reliability of observations can be improved by making sure that behavioural categories have been properly operationalised, and that they are measurable and self-evident. For instance, the category ‘pushing’ is much less open to interpretation than ‘aggression’. Categories should not overlap for example, ‘hugging’ and ‘cuddling’ and all possible behaviours should be covered on the checklist. If the categories are not operationalised well, are overlapping, or absent different observers have to make their own judgements of what to record where, and may well end up with differing inconsistent records.
improving reliability: experiments
Control: In lab experiments, the researcher can control many aspects of the procedure, including the instructions given to participants and the conditions they are tested in.
Replicability: This high level of control makes experiments more replicable, meaning that if someone repeats the experiment, they should obtain similar results.
Lab vs. Field: This control is easier in labs compared to field studies, where external variables are harder to manage, and researchers have less control over the situation.
Precision vs. Reliability: The reliability in a lab setting focuses on precise replication of methods rather than the reliability of a specific finding.
Variability: Slight differences in conditions across tests might still affect the reliability of the findings.
