Module 1 - Measuring Human Behaviour

Question 1

Sources of Error

Answer 1

Random error or bias. Can be form observers, researcher or participant

Question 2

Reliability

Answer 2

The consistency or repeatability of measures

Question 3

Validity

Answer 3

The accuracy of what we are trying to measure

Question 4

Reliabilty & Validity

Answer 4

Validity rides on the back of reliablilty, but reliability does not guarantee validity.
Not reliable would show measurements that are very different from eachother. Not valid measurements would show a score that is not a comprehensible.

Question 5

Testing Reliability - Inter-Rater or Inter-Observer

Answer 5

Used to assess the degree to which different raters/observers give consistent estimates of the same phenomenon - agreement between the scores of two or more independent observers or judges. Very important when measure are subjective.

Question 6

Testing Reliability - Test-Retest

Answer 6

Used to assess the consistency of a measure from one time to another

Question 7

Testing Reliablilty - Parallel-Forms

Answer 7

Used to assess the consistency of the results of two tests constructed in the same way from the same content domain.
* Split-half reliability
* Item-total correlation

Question 8

Testing Reliability - Internal Consistency

Answer 8

Used to assess the consistency of results across items within a
test
Cronbach’s alpha – the average correlation among all possible
pairs of items (.80 or above)

Question 9

Validity is…

Answer 9

Truth & accuracy. Are we justifying our conclusions? do the findings reflect reality? Are we measuring what we intend to measure?

Question 10

Construct Validity

Answer 10

A “construct” refers to a behaviour or process we are intersted in studying, for example; depression or short-term memory

Question 11

Construct Validity Manipulations can be…

Answer 11

Instuctional - conditions defined by what you tell participants
Environmental - stage of events, present a stimulus, induce a state
Stooges - fake participants used to alter conditions

Question 12

Construct Validity - Convergent

Answer 12

Related to the degree to which the measure converges on other constructs that is theoretically should be similar to.
Do scores on the measure correlate with scores on other similar measures related to the construct?

Question 13

Construct Validity - Disciminant (Divergent)

Answer 13

Relates to the degree to which the measure diverges from other constructs that it should be not similar to
Do scores on the measure have low correlations with scores on
other different measures that are unrelated to the construct?

Question 14

Construct Validity - Face Validity

Answer 14

On face value, does the measure seem to be a good translation of the construct? If you ask the participants, doe sit make sense as a measure of arithmic ability? One way to achieve this is to ask experts in the field.

Question 15

Construct Validity - Content Validity

Answer 15

Does the measure assess the entire range of characteristics that are representative of the contruct it is intending to measure?

Question 16

Construct Validity - Criterion Validity

Answer 16

Concurrent - Do tscores on the measure distinguish participants on other variables that we would expect to be related to it? (Eg; Depressives from non-depressives)
Predictive - Are scores on the measure able to predict future outcomes? (eg; attitudes, behaviours, performance)

Question 17

Construct Validity Manipulations - Doing it right

Answer 17

• Reduce random error (replicate procedure)
• Reduce experimenter bias
• Reduce participant bias
• Ensure manipulation has contruct validity
• Do a manipulation check - ask participants aboiut various aspects, beliefs attitudes etc.,

Question 18

External Validity

Answer 18

The extent to which the results can be generalised to other relevant populations, settings or times
Studies that have good external validity when results can be replicated;
- Using alternative operationalisation of variables
- Measuring a different sample of particiapnts
- Conducting the research in another setting

Question 19

External Validity - Ecological

Answer 19

The extent to which the results can be generalised to real-life settings

Question 20

External Validity - Population Generalisation

Answer 20

Applying the results from an experiment to a group of participants that is different and more encompassing than those used in the original experiment

Question 21

External Validity - Environmental Generalisation

Answer 21

Applying the results from an experiment to a situation or environment that differs from that of the original experiment

Question 22

External Validity - Temporal Generalisation

Answer 22

Applying the results from an experiment to a time that is different from the time when the original experiment was conducted

Question 23

Internal Validity

Answer 23

Ability to draw conclusions about casual relationships from the results of the study
The extent to which we can say that any effects on the DV were caused by the IV
Eliminations of the alternative explanations for the observed relationships.
Strong internal validity requires analysis of these three elements; co-variation, tempral presedence, elimination of alternative explanations

Question 24

Threats to Internal Validity - Selection Bias

Answer 24

A threat to internal validity that can occur if participants are chosen in such a way that the groups are not equal before the experiment
Differences after the experiment may reflect differences that existed before the experiment began
Differences after the experiment may reflect differences that existed before the experiment began plus a treatment effect

Question 25

Threats to Internal Validity - Maturation

Answer 25

Changes in participants during the course of an experiment or between measurements of the DV due to the passage of time
Permanent – e.g., age, biological growth, cognitive development
Temporary – e.g., fatigue, boredom, hunger
Most common is naturally occurring developmental processes (i.e., children)

Question 26

Threats to Internal Validity - Statistical Regression - Regression Towards the Mean

Answer 26

Participants with extreme scores on the first measurement of the DV tend to have scores closer to the mean on the second measurement of the DV
Subsequent scores are still likely to be extreme in the sames
When you have extreme scores, it is difficult to maintain that degree of extremity over repeated measures
If participants are selected on the basis of extreme scores – regression to the mean is always going to be a possible explanation for higher or lower scores on a repeated (or similar) test

Question 27

Threats to Internal Validity - Mortality/Attrition

Answer 27

Relates to premature dropouts and differential dropout across experimental conditions
If a differential dropout rate occurs, then it is likely that the groups of participants are not as equal at the end of the experiment as they were before
If the intervention is unpleasant or very demanding or is not working – dropout could be higher in this group than in the control group
Or if the participants in the control group see no benefit or no improvement – dropout could be higher in this group than the experimental group

Question 28

Threats to Internal Validity - History

Answer 28

Outside events that may influence participants in the course of the experiment or between the DV measurements in a repeated-measures design
History can include major historical events like terrorist attacks, political changes, or smaller personal changes like joining a gym or changing jobs or having a baby
If they are relevant to the study in some way, these events can influence the second DV score obtained – independently of any intervention

Question 29

Threats to Internal Validity - Testing

Answer 29

Prior measurement of the DV may influence the results obtained for subsequent measurements of the DV
Measuring the DV can cause a change in the DV
E.g., participant become aware of study aims

Question 30

Threats to Internal Validity - Practice Effect

Answer 30

When a beneficial effect on a DV measurement is caused by previous experience with the DV measurement itself
E.g., making free-throws

Question 31

Threats to Internal Validity - Instrumentation

Answer 31

Changes in measurement of the DV that are due to the measuring device (equipment or human)
The equipment or human measuring the DV changes the measuring criterion over time
What sort of things could that include?
How could we try to manage this?

Question 32

Threats to Internal Validity - Effects of studying people - Observee reactivity (Hawthorne Effect)

Answer 32

Participants change their behaviour when they know they are being observed (“reactivity”)
When might we see this happening?
What can we do to try to counteract this?

Question 33

Threats to Internal Validity - Effects of studying people - Social desirability

Answer 33

Reporting inaccurately on sensitive topics in order to present in the best possible light
When might we see this happening?
What can we do to try to counteract this?

Question 34

Threats to Internal Validity - Demand Effects

Answer 34

Relate to an aspect of the research that allows participants to guess what the research is about

Question 35

Threats to Internal Validity - Placebo Effects (a type of non-specific treatment effect)

Answer 35

Results from participant’s own expectations about experiments or expectations about what will happen or what is meant to happen

Question 36

Threats to Internal Validity - Experimenter Bias

Answer 36

Errors in a research study due to the predisposed notions or beliefs of the experimenter e.g., Observer bias (selective viewing or interpretation of behaviours, NLP)

Question 37

Controlling threats to Internal Validity

Answer 37

• Randomly allocate participants to levels of the IV(‘s)
• Treat all conditions equally except for intended IV manipulations
• Use appropriate control conditions where relevant
• Use double-blind studies where possible

Question 38

Replication Crisis in Psychology

Answer 38

• 97% of original studies had had significant effects
• Only 36% of replications were significant
• Replication mean effect size - half of the original studies
• Rates varied across fields within Psychology
• Cognitive highest (50%) Social lowest (25%)
• This does not necessarily mean that the original studies were “wrong”

Question 39

Replication Crisis in Psychology - Why does this happen?

Answer 39

Replications are uncommon
In the top 100 psych Journals between 1900- 2012, only 1.6% were replications (Makel et al., 2012)
Substantial bias towards publishing significant findings (and not null findings)
Times change – especially in terms of areas like social psych
Alpha cutoffs are arbitrary

Question 40

Replication Crisis in Psychology - What has been the response of the Psychology field

Answer 40

Increase in replications
Pre-registration of studies (if methods are good they will be published regardless of significance)
Suggestion that we might consider probability based analyses rather than null hypothesis significance testing

Question 41

Module Summary

Answer 41

What is measurement error and where does it come from?
Define reliability and validity, and how they are related to each other
How can we measure how reliable a measure is?
What are the key types of validity?
Describe the types of validity, and the threats they might face