Reliability, Validity & Inferential Statistics

Question 1

What is reliability?

Answer 1

whether something is consistent in a study and whether it is replicable

Question 2

What is external reliability?

Answer 2

the consistency of measures over time, can be assessed using the test-retest method

Question 3

What is test-retest reliability?

Answer 3

involves presenting the participants with the same test/questionnaire on two separate occasions and seeing whether there is a positive correlation between the two

Question 4

What is inter-observer (or inter-rater) reliability?

Answer 4

the extent to which two or more observers are observing and recording behaviour in the same way - due to the issue of subjectivity in observational studies

Question 5

What is internal reliability?

Answer 5

refers to how consistently a method measures within itself - may use the split-half technique to see how similar they are

Question 6

What us the split-half technique?

Answer 6

half of the scores in a test are compared with the other half to see how similar they are - may be done by comparing odd scores on IQ test to the even scores, comparison seen by looking at how well the scores correlate

Question 7

How do we improve reliability in an experiment?

Answer 7

• controlling variables e.g. given standardised instructions as they can check the replicability of the findings

Question 8

How do we improve reliability in observations?

Answer 8

• observers must have clear and operationalised categories of behaviour and also trained in how to use the categories (may be shows video footage of similar behaviour they will observe)

Question 9

How do we improve reliability in content analyses?

Answer 9

• use very tightly controlled and defined coding units and training beforehand

Question 10

How do we improve reliability in questionnaires?

Answer 10

• identify questions causing issues and rewrite/replace these questions to avoid the problem of subjective interpretation changing how they’re responded to
• in general, closed-questions more reliable than open-questions

Question 11

How do we improve reliability in interviews?

Answer 11

• use same interviewer each time

- structures more reliable than unstructured

Question 12

What is validity?

Answer 12

whether something is true, and if it measures what it’s set out to measure

Question 13

What is the internal validity?

Answer 13

in relation to experiments, whether the results were due to the manipulation of the IV rather than other factors such as extraneous variables or demand characteristics

Question 14

What is the external validity?

Answer 14

whether it is possible to generalise the results beyond the experimental setting

Question 15

How do we asses the internal validity?

Answer 15

• face validity

- concurrent validity

Question 16

What is the face validity?

Answer 16

simple way of assessing whether a test measures what it claims to measure which is concerned with face value - e.g. does an IQ test look like it tests intelligence

Question 17

How is face validity assessed?

Answer 17

by giving the test to experts in the field to look over, as they will know what is more likely to be relevant
- problem with it is that it is simply an opinion and there is no guarantee that it is correct

Question 18

What is the concurrent validity?

Answer 18

comparing a new test with another test of the same thing to see if they produce similar results, if they do then the new test has concurrent validity - if there is a strong positive correlation (i.e. exceeds 0.80+) between participants’ scores on the two tests

Question 19

What is ecological validity?

Answer 19

the extent to which findings of a research study are able to be generalised to real-life settings

Question 20

What is temporal validity?

Answer 20

refers to how likely it is that the time period when a study was conducted has influenced the findings and whether they can be generalised to other periods in time

Question 21

How do we improve the validity in experiments?

Answer 21

• having a control group means the effect on the DV of the change in IV can be seen by comparing the results from the experimental condition to those from the control where the IV did not change
• single blind procedures used to prevent the participants’ knowledge of which condition they are in from affecting their behaviour
• double blind procedures
• deception
• standardised instructions
• repeated measures design OR independent groups allocate to random conditions
• with repeated measures, use counterbalancing to reduce order effects

Question 22

How do we improve the validity in observations?

Answer 22

• behavioural categories need to be tightly operationalised so the observations are more objective
• using covert observations
• using naturalistic observations

Question 23

How do we improve the validity in self-reports?

Answer 23

• validity of these threatened by participants not telling the truth due to social desirability bias or not understanding the questions properly
• interviews mean participants can ask for the questions to be clarified
• questionnaires decrease social desirability bias (researcher does not need to be present) especially if responses are kept anonymous
• check for lies using a lie scale

Question 24

What is a lie scale?

Answer 24

same thing is asked about more than once, but in different ways and if answers are inconsistent this indicates participants have either misunderstood or lied

Question 25

How do we improve the validity in qualitative methods?

Answer 25

• triangulation

Question 26

What is triangulation?

Answer 26

the use of a number of different sources as evidence, if all of these sources agree, interpretation is more valid

Question 27

What do we need to consider when deciding which inferential test to use?

Answer 27

• design
• difference
• data (level of measurement)