4: Validity Reliability

Question 1

What contributes to soundness of experiments?

Answer 1

precision

accuracy

sensitivity

specificity

reliability

validity

Question 2

what is precision?

Answer 2

Consistency, reliability, homogeneity of the data

Question 3

what is accuracy?

Answer 3

Based on the precision of the measurements. Good accuracy if averaged values ≈ standard. Standards are rare in the biological sciences (behavioural or neurosciences). In psychophysics (Signal Detection Theory), accuracy = specificity + sensitivity

Question 4

what is sensitivity?

Answer 4

Measures should be sensitive enough to detect differences in a characteristic that are important to the investigator

Question 5

what is specificity?

Answer 5

Measures should be specific to the characteristics, group, phenomenon, etc. investigated

Question 6

what is reliability?

Answer 6

Consistency of the measures. Precision

Question 7

what is validity?

Answer 7

Does a variable represent what it is intended?

Question 8

what are nuisance variables?

Answer 8

We hinted at this before with the concept of “distortion”

Two types:
Systematic error or bias
Random error (or error variance)

Question 9

what is influenced by Systematic error or bias?

Answer 9

accuracy

Question 10

what type of variables are a source of bias?

Answer 10

extraneous/ confounded variables

Question 11

Types of biases/systematic errors and their solutions

Answer 11

Observer/Experimenter»>
Blinding procedures

Subject/Participant&raquo_space;> Blinding/ Unobtrusive measures

Apparatus&raquo_space;> Calibration

Question 12

what is random error?

Answer 12

The precision of measurements (and therefore the consistency and reliability) is influenced by random error unpredictable fluctuations)

Question 13

sources of random error

Answer 13

Random error is due to random fluctuations in participants, experimental conditions, methods of measurement, etc.

Main sources of random error:
*Observer / experimenter reliability
*Participant / subject reliability
*Instrument / apparatus reliability

Question 14

contributors of precision

Answer 14

Calibration of an apparatus

Consistency of a participant

Environmental and other factors

Archery example: Bow/sight, archer, wind

Question 15

How to assess precision?

Answer 15

Measures of variability (descriptive statistics)

measures of concordance

Question 16

what are measures of variability

Answer 16

Standard deviation (sd) of repeated measurements.

Coefficient of variation (cv): (sd ÷ mean) × 100

Question 17

what are measures of concordance?

Answer 17

Correlation coefficient: Consistency of results of paired
measurements. The coefficient of correlation is an index of
concordance

Question 18

what is reliability?

Answer 18

Consistent results over repeated measurements. Reliability refers to the PRECISION of your measures

Question 19

assessment methods

Answer 19

Test-retest reliability/consistency: Stability of test scores over time.

Alternative (parallel) forms reliability/consistency: e.g., recognition/recall example with
tests.

Internal consistency: How consistent is the measure across items intended to measure the same concept, e.g., split-half reliability/consistency » use of two lists in a memory test.

Inter-rater reliability: see next slide.

In some cases: Intra-rater reliability

Question 20

what is Inter-observer/rater consistency or reliability?

Answer 20

consistency of recording
and scoring between ALL OBSERVERS

with an inter- observer reliability measure such as an index of concordance, kappa coefficient, Kendall coefficient

Question 21

what is Intra-observer/rater consistency or reliability?

Answer 21

each observer, individually, records,
interprets or identifies SIMILAR behaviours or events the SAME
WAY.

with an intra- observer reliability measure)

Question 22

what does intra mean

Answer 22

within

Question 23

what does inter mean

Answer 23

between

Question 24

what is validity about?

Answer 24

Validity is about the threats to valid inference making; Is the procedure you chose measures what it is intended to measure?

Question 25

what are the four main types of threats to validity

Answer 25

construct validity

statistical conclusion validity

internal validity

external validity

Question 26

what is construct validity

Answer 26

The wrong independent variables are identified

Question 27

what is statistical conclusion validity

Answer 27

random error and wrong selection of statistical tests, low power, violation of statistical assumptions, fishing, etc

Question 28

types of validity of a measurement

Answer 28

face validity

content validity

construct validity

criterion validity

Question 29

what is face validity

Answer 29

How well the test appears to measure what it is designed to measure. It is a plausible measure of the variable we want to estimate. Face value. Non-scientific. E.g.: Common sense definition of stress.

Question 30

what is content validity

Answer 30

How adequately the measure addresses the representativeness of the measured event or phenomenon as a whole (i.e., represents the whole content). Expert opinion can determine this type of validity. E.g., you lack content validity if you want to measure stress, but you only take behavioural measures and no physiological measures (or vice versa)

Question 31

what is construct validity

Answer 31

A measure of how well a test and operational definition assess some underlying (theoretical) construct or variable. Depends heavily on the operational definitions, e.g., “stress”. The measurement procedure and the variable it measures are in agreement. An assay of glucocorticoids suggesting high levels of cortisol is associated with high stressful situations

Question 32

what is criterion validity

Answer 32

The ability of a measure to assess (or predict) an outcome or criterion. Performance measures

Question 33

what are the subtypes of criterion-related validity

Answer 33

concurrent validity

convergent/divergent validity

discriminant validity

predictive

Question 34

what is concurrent validity

Answer 34

A measure of how well an assay estimates a criterion/ performance in relation to another (concurrent) phenomenon or group of subjects at the same point in time. A new test or assay is validated as it concurs with an older, better established one.

Question 35

what is convergent/ divergent validity

Answer 35

Two or more methods of measurement converge/diverge upon one another. Strong relationship between the scores are found. Can be established by correlation

Question 36

what is discriminant validity

Answer 36

The methods of measurement diverge upon one another and the divergence is expected. A measure of stress should not be expected to be highly correlated with a measure/construct of empathy

Question 37

what is predictive validity

Answer 37

A measure of how well an assay predicts a phenomenon on a time criterion: e.g., pre/post. Measures predicts future states.

Question 38

what is the relationship between validity and reliability

Answer 38

A measure can have high reliability but not low validity.

A measure cannot be more valid than it is reliable

Question 39

types of validity and what they mean

Answer 39

Internal validity

External validity

Question 40

what is internal validity and what does that mean

Answer 40

Measures what it is supposed to?

Associated with the criteria for ultimate (analytic) experiments (i.e., fully “experimental”).

• No confounded variables
• Controlled variables… are controlled
• Appropriate control group(s)
• Random assignment (randomization)
• Random selection (sampling) ~ preferable, but rarely attained

Question 41

what is external validity and what does that mean

Answer 41

Generalization potential or generalizability

Generalizability of the data!
* Species
* Environments
* Cultures
* Age groups
* Conditions, etc.

Determines the applications and implications of an experiment.

Question 42

what is the criteria for external validity

Answer 42

population selection

operational definitions

parameter values

demand characteristics

Question 43

what is population selection

Answer 43

Converging evidence (from different populations) and representativeness of the sample

Question 44

what are operational definitions

Answer 44

Agreement on definitions. For example, “stress”

Question 45

what are parameter values

Answer 45

The values you select for each variable in your experiment should be well defined. Applies to control variables and independent variables

Question 46

what are demand characteristics

Answer 46

Cues in a research procedure that influence the behaviour of subjects are absent or minimized.

Have the potential to influence internal validity

Question 47

what is ecological validity ( case of external validity)

Answer 47

Related to external validity (generalizes/applies well to other people, settings, conditions, etc.).

Are experiments done in the laboratory generalizable to the “real world”?

Not a central concern of neuroscience (in general). Technological limitations
and constraints.

Question 48

what are mediator variables

Answer 48

Provides a causal link in a sequence between an IV and a DV. Answers the WHY?

Question 49

what are moderator variables

Answer 49

Modulates the strength or direction of the relation between an IV and a DV. Answers the WHEN, and for WHOM or WHAT?