Week 9-Reliability and Validity

Question 1

What Makes a Good Experiment?

Answer 1

■ Reliability.
■ Sensitivity.
■ Validity.

Question 2

What’s Reliability?

Answer 2

■ Consistency/dependability of a measure.
■ The ability of a measure to produce the same or similar results on repeated administrations.
■ Is it any good at measuring?
■ Internal reliability
■ External Reliability

Question 3

What’s Internal Reliability?

Answer 3

■ The extent to which a measure is consistent
within itself.
■ Split-half reliability:
– Compares the results of one half of a test with the other half.
– Split the items into two groups and correlate
them.
– A participant scoring high on one half, should
score high on the other half.

Question 4

But what if all the good questions were at the beginning, rubbish questions at the end?

Answer 4

■ May give a weaker correlation than is fair.
■ Any correlation might just be down to chance, depending on how the questionnaire was split.

Question 5

What’s Cronbach’s alpha?

Answer 5

■ A test that splits items equally in every way
possible.
■ It then correlates ALL halves with ALL other halves
■ Much more robust measure of internal reliability.
■ Should have a high correlation coefficient >.70
■ SPSS also gives information about how each item
correlates with all other items.
■ And how alpha would change if an item was
removed.
■ The higher the alpha – the more reliable the questionnaire.

Question 6

What’s External Reliability?

Answer 6

■ The extent to which a measure varies from one use to another.
■ Test-retest reliability:
– the stability of a test over time.
– A good test is consistently reliable.
– Administer the test now, then give the same test later to the same participants
– A good test will have a high correlation.

Question 7

What’s inter-rater reliability (external reliability)

Answer 7

– Usually used in observational studies
– Degree to which different raters give consistent estimates of the same behaviour.
– Correlation to check reliability.
■ To improve inter-rater reliability:
– Clear categories/definitions.
– Training.

Question 8

How can you improve reliability in general?

Answer 8

– Improve quality of items.
– Increase/decrease number of items.
– Increase sample size.
– Choose appropriate sample.
– Control conditions.

Question 9

What’s sensitivity?

Answer 9

■ Detecting even a small effect of the IV on the DV.
– Large sample size
– Sizeable/varied effects
– Control variability

Question 10

What’s the benefits of sensitivity?

Answer 10

■ Not too hard or too easy.
■ Wide range of scores.
■ E.g. The effect of high levels of alcohol on reading ability?
– IV = alcohol intake.
– DV?
– Ask participants if they found the words hard to
read - Yes/No.
– Time them reading the word list.
■ Properties of your sample:
– Facebook/instagram usage – students or
pensioners?

Question 11

What’s Validity?

Answer 11

■ A test/measure.
■ A study/experiment.
■ Test validity - the ‘truthfulness’ of a measure in that it measures what it claims to.
– Content validity.
– Construct validity.
– Criterion validity.

Question 12

What’s content validity?

Answer 12

– Face validity - whether the test appears to measure what it claims to.
– Purpose is obvious.
– Social desirability?

Question 13

What’s construct validity?

Answer 13

The degree to which a test measures the construct/psychological concept at which it is aimed.
– Convergent validity
– The degree to which it correlates with other
measurements assessing the same construct.
– Divergent/Discriminant validity
– The degree to which it does not correlate with
other measurements assessing different concepts.

Question 14

What’s the criterion validity?

Answer 14

Whether a test reflects a certain set of abilities i.e. the degree to which a measurement can accurately predict specific criterion variables.
– Concurrent validity:
– How well a test correlates with a previously
validated measure, given at the same time.
– Predictive validity:
– How well it predicts future performance.

Question 15

The validity of a study/experiment:Types

Answer 15

– External validity
– Internal validity
– Ecological validity

Question 16

What’s external validity?

Answer 16

– The extent to which the results of a study can be generalised to different populations, settings and conditions.
 E.g. Which styles of leadership work best in
a hostile military situation?
 Test different styles in military training war
games.
 How like real combat are military training
exercises?

Question 17

What are some aspects of high external validity?

Answer 17

– Extend to new people/situations.
– Measure the construct that it claims to measure.
– Use a representative sample.
– Replicate using a new group.

Question 18

What is internal validity?

Answer 18

– When we can be confident that manipulating the IV affects the DV.
■ Main goal of experimental research?
■ Causation - 3 criteria:
– Show co-variation/correlation.
– Show time-order relationships.
– Eliminate all other possible causes.
■ Free of confounding = internally valid.

Question 19

What are the 5 key threats to internal validity?

Answer 19

– Testing intact groups (i.e. non-random allocation).
– Order/practice/fatigue/transfer.
– Presence of extraneous variables.
– Unequal loss across groups.
– Expectancy effects/demand characteristics.

Question 20

How do you test intact (natural) groups?

Answer 20

– Between-subjects design only.
– Cannabis and memory.
– Psychology vs. Philosophy students.

Question 21

What are order/practice/fatigue/transfer effects?

Answer 21

Within-subjects design only.
– Order/practice effects:
■ Better scores in 2nd/3rd etc.
– Fatigue/boredom effects:
■ Poorer scores in 2nd/3rd etc.
■ Overcome by counter-balancing.

Question 22

What are differential transfer effects?

Answer 22

■ The effects of one condition affect participants’ performance in subsequent conditions.
E.g. Mnemonic learning techniques.
E.g. Mogg et al. (2008).
– Overcome by using a between-subjects design or a within-subjects design with each condition sufficiently spaced.

Question 23

What are extraneous variables?

Answer 23

– May cause variation in scores but are NOT part of the experimental manipulation.
 Which experimenter conducted the experiment.
 Room temperature.
 Trial order.
 Assessment tools/techniques.

Question 24

How do we deal with extraneous variables?

Answer 24

– Control everything you possibly can!
– E.g. Same experimenter, constant room temperature etc.
– Counter-balance everything.
– Standardised assessment tools/techniques.

Question 25

What’s unequal participant loss across groups?

Answer 25

– Most experiments suffer from participant drop out.
– Only a problem in between-subjects designs.

Question 26

What are the different types of expectancy effects?

Answer 26

■ Expectancy effects:
– Participant and/or experimenter.
■ Participant expectancy effects:
– E.g. Cannabis and memory.
– Placebo.
■ Experimenter expectancy effects:
E.g., Rosenthal and Fode (1963)

Question 27

What are the participant and experimenter expectancy effects?

Answer 27

– Double-blind procedure - Neither know whether it’s drug or placebo.
– Expectancies of both held constant across conditions.

Question 28

What are the demand characteristics?

Answer 28

– Cues/information used to guide behaviour
- what they believe the experimenter wants/ expects them to do.
– Rumours of the study.
– Setting.
– Explicit or implicit communication.

Question 29

What can demand characteristics be overcome by?

Answer 29

– Deception.
– Double-blind procedures.
– Unobtrusive manipulations/measures.
– Between-subjects design.

Question 30

What’s balance?

Answer 30

■ Experimental design = control =  internal validity.
■ ‘Real world’ DV =  external validity.
■ Ask participants to remember to do something at a particular point in the study as well as recall a list of words.
■ Multi-method approach:: Conduct a controlled
experiment and a naturalistic study.
■ Same conclusions - convergent validity.

Question 31

What’s ecological validity?

Answer 31

– How much a method of research resembles ‘real life’.
– Are the results generalisable across different settings?
– Lab based = artificial.
– Representative sample.