Reliability and Validity

Question 1

What is reliability

Answer 1

Reliability refers to the consistency/repeatability of results of a measurement. “How” reliable something is relative and depends on the situation.

Question 2

Types of reliability

Answer 2

• Observers: Inter-Observer reliability
• Observations: Internal (Split-half) reliability
• Occasions: Test-retest reliability

Question 3

What is inter-observer reliability?

Answer 3

Inter-observer reliability is the degree to which observers agree upon an observation or judgement
• Can be frequency or categorical judgement

Question 4

How is inter-observer reliability tested?

Answer 4

Measure by looking at shared relationships between observers (i.e. correlation). Cohen’s kappa, Pearson’s correlation coefficient (r), etc. depending on whether continuous or discrete

Question 5

What is internal reliability?

Answer 5

Internal reliability is the degree to which specific items/observations in a multiple item measure behave the same way. I.e. are they measuring the same thing?

Question 6

How is internal reliability tested?

Answer 6

Tested by dividing test into 2 halves, then looking at correlation between them. If it has high internal reliability, an individual’s performance on the first half should correlated with the second half.

Question 7

What is test-retest reliability?

Answer 7

Test-retest reliability is the extent to which scores on a test/measure remain stable over time.

Question 8

What is validity?

Answer 8

Validity refers to how well a measure or an

operationalised variable corresponds to what it is supposed to measure/represent.

Question 9

Types of validity

Answer 9

• Internal validity
• External validity
  
  • Population validity
  • Ecological validity
• Construct validity
• Content validity
• Criterion validity

Question 10

Internal validity

Answer 10

• How convincing is the evidence for causality in a study/series of experiments?
• i.e. how strong is the inference that the independent
variable and the dependent variable are causally
related?

Question 11

J.S. Mill: 3 requirements to establish causality

Answer 11

• Covariation
• Temporal Sequence
• Eliminating confounds (rival explanations/hypotheses)
  
  • Third-variable problem

Question 12

Equifinality

Answer 12

Another threat to internal validity. Most things have multiple causes .

Question 13

External validity

Answer 13

How well does a causal relationship
hold across different people, settings,
treatment variables, measurements
and time. 
Two types: Ecological and population validity.

Question 14

Population validity

Answer 14

• Making cross-cultural inferences from Western, Educated, Industrialized, Rich, Democratic samples?
• Differences in tasks ranging from motivation, reasoning and even visual perception
  
  • Muller-Lyer Illusion: Americans vs. the San people of the Kalahari

Question 15

Ecological Validity

Answer 15

How well do results of laboratory experiments generalise to real-life settings?
• E.g. aggression studies in the lab vs. in real life
• Bandura (1961, 1963): Bobo doll experiment

Question 16

Construct validity

Answer 16

How well do your operationalized variables
(independent and/or dependent) represent the
hypothetical or abstract variables of interest

Question 17

Content validity

Answer 17

Degree to which the items or tasks adequately sample the
target domain
• i.e. how well does a measure/task represent all the facets of a construct

Question 18

Criterion validity

Answer 18

To what extent can a procedure be used to infer or predict some criterion (outcome)
Two types: concurrent and predictive

Question 19

Concurrent validity

Answer 19

A type of criterion validity - to what extent can a procedure be used to infer some criterion

Question 20

Predictive validity

Answer 20

A type of criterion validity - to what extent can a procedure be used to predict some criterion

Question 21

When do person confounds occur?

Answer 21

Person confounds occur when a variable seems to cause something because people who are high or low on this variable also happen to be high or low on some individual difference variable (e.g. demographics characteristic) that is associated with the outcome variable of interest.

Question 22

When do operational confounds occur?

Answer 22

Operational confounds occur when a measure designed to assess a specific construct such as depression, memory, or foot size inadvertently measures something else as well.

Question 23

What is a confound?

Answer 23

A threat to internal validity, that undermines a causal explanation

Question 24

What is an artifact?

Answer 24

A threat to external validity, a by-product of testing procedure or sample that biases all results.
Unlike confounds, artifacts stay constant and are present in all groups

Question 25

What are some artifacts?

Answer 25

Hawthorne effect, history effect, and selection bias/non-response bias

Question 26

What is the Hawthorne effect?

Answer 26

That mere act of measurement changes the nature of something | - a form of participant reaction bias

Question 27

Advantages to Surveys?

Answer 27

Quick and efficient Large samples Obtain public opinion almost immediately Easy to use

Question 28

Limitations of surveys?

Answer 28

``` Non-random sampling Haphazard samples (availability and volunteer bias) Question wording effects ```

Question 29

Advantages of naturalistic observation?

Answer 29

Allows the study of issues not amenable to experimentation | Useful in the initial stages of investigation

Question 30

Limitations of naturalistic observation?

Answer 30

Low internal validity Time consuming Hawthorne effect - threat to external validity

Question 31

Advantages of participant observation?

Answer 31

It can be used in situations that otherwise might | be closed to scientific investigation

Question 32

Limitations of participant observation?

Answer 32

The dual role of the researcher maximizes the chances for the observer to lose objectivity and allow personal biases to enter into the description

Question 33

Advantages of longitudinal studies?

Answer 33

``` • Genuine changes and stability of some characteristics observed • Major points of change observed • Temporal sequence • Minimise age-cohort effects ```

Question 34

Disadvantages of longitudinal studies?

Answer 34

* Time consuming and expensive | * Participant attrition – threat to validity

Question 35

Advantages of cross-sectional studies?

Answer 35

• Relatively inexpensive and less time consuming • Low attrition rate

Question 36

Disadvantages of cross-sectional studies?

Answer 36

* Cannot observe changes in individuals * Insensitive to abrupt changes * Age-Cohort effects

Question 37

What are age-cohort effects in cross-sectional studies?

Answer 37

Cohort differences are confounded with age differences. For example, measuring computer skills at different age points - Different exposure to computers

Question 38

What is experimental mortality also known as? What's an example of a situation in which it is a problem?

Answer 38

Experimental mortality is also known as heterogenous attrition. For example, in study on antipsychotics and schizophrenia, experimental group receive antipsychotics which produce side effects and also reduce potentially pleasurable symptoms such as schizophrenia. Hence, more drop out of the experimental group. This becomes a threat to internal validity.

Question 39

Individual difference confounds include...

Answer 39

Experimental mortality | Participant Reaction Bias (participant expectancies, participant reactance, evaluation apprehension)