Week 3 - Reliability and Validity

Question 1

Classical Test Theory

Answer 1

Test scores are result of

• Factors that contribute to consistency
• Factors that contribute to inconsistency (characteristics of test takers, things that have nothing to do with attribute such as situation, environment)

Question 2

X = T + e

Answer 2

X = obtained score
T = true score
e = errors of measurement

Question 3

Sources of Error

Answer 3

• Item selection
• Test administration
• Test scoring
• Systematic measurement error

Question 4

Domain-sampling mode

Answer 4

a way of thinking that sees the test as a representative sample of a large domain of possible items that could be included on the test

• considers the problem of using only a sample of items to represent a construct
• as test gets longer, should represent construct better, increase reliability

Question 5

Inter-rater reliability

Answer 5

the extent to which different raters agree in their assessments

Question 6

Method variance

Answer 6

the variability among scores that arises because of the form as distinct from the content of the test - the method of administering the test

Question 7

Reliability

Answer 7

the consistency that a test will give the same result each time it is used to measure the same thing

Question 8

Stability over time

Answer 8

the extent to which test scores remain stable when a test is administered on more than one occasion

Question 9

Internal consistency

Answer 9

the extent to which a psychological test is homogeneous or heterogeneous

Question 10

Social desirability bias

Answer 10

a form of method variance that arises when people respond to questions that place them in a favourable or unfavourable light

Question 11

Test-Retest Stability

Answer 11

The same test administered to the same group twice at different points
- may not get identical scores due to practice effects, maturation, treatment effects or setting

Question 12

Parallel or alternate forms of reliability

Answer 12

Two forms of same test developed, different items selected according to the same rules

Parallel - same distribution of scores (mean and variance equal)
Alternate - different distribution of scores (mean and variance may not be equal)

Both matched for content and difficulty

Question 13

Split half method

Answer 13

Test is divided into halves that are compared

- useful in overcoming logistical difficulties of test-retest reliability

Question 14

Measuring Internal consistency

Cronbach’s Alpha

Answer 14

Cronbach’s alpha - a generalised reliability coefficient for scoring systems that are graded (i.e agree - disagree)

Acceptable levels of reliability

• .70-.80 acceptable or good
• greater than .91 may indicate redundancy

Question 15

Standard error of measurement (SEM)

Answer 15

allows estimation of precision of an individual test score
- the larger the SEM, the less certain we are that the test score represents the true score

Reliability coefficient (r) - an index of the ratio of true score to error score variance in a test
- SEM = (1-r)

Question 16

Validity

Answer 16

the extent to which a test measures the construct it is intended to measure
- inferences from test must be appropriate, meaningful, and useful

Question 17

Face validity

Answer 17

does the test look like it measures the relevant construct

Question 18

Content validity

Answer 18

the extend to which items on a test represent the universe of behaviour the test was designed to measure

• logical deduction rather than strict analysis

Question 19

Construct underrepresentation

Answer 19

failure to capture important components of a construct

Question 20

Construct-irrelevant variance

Answer 20

measuring things other than the construct of interest

Question 21

Criterion related validity

Answer 21

the extent to which a measure is related to an outcome
Good criterions share - reliable, appropriate

relationship between test and criterion usually expressed as correlation

Question 22

Predictive evidence

criterion related validity

Answer 22

how well the test predicts performance on a criterion

Question 23

Concurrent evidence

criterion related validity

Answer 23

refers to a comparison between the measure in question and an outcome assessed at the same time

Question 24

Incremental validity

Answer 24

the extent to which knowledge of a score adds to that obtained by a pre-existing test score or psychological characteristic

Question 25

Construct validity

Answer 25

concerned with establishing how well a test measures a psychological construct

Question 26

Convergent evidence 
(construct validity)

Answer 26

refers to the degree that two constructs which should be related, are related
- identify relationships we would expect if the test is actually measuring the construct

Question 27

Discriminant (divergent) evidence

construct validity

Answer 27

aims to demonstrate that the test is unique

- low correlations should be observed with constructs that are unrelated to what the test is trying to measure

Question 28

Factor analysis
(construct validity)

Answer 28

• to observe patterns
• items may cluster, which is attributed to action of latent/unobserved variables or factors
• Exploratory factor analysis
• Confirmatory factor analysis

Question 29

Decision-theoretic approach to predictive validity

Answer 29

includes

• cutting point
• valid positive and negative decisions
• false positive and negative decisions
• base rate
• selection ratio

Question 30

Cutting point

Answer 30

the test score or point on the scale that is used to split those being assessed into two groups predicted to show and not show the studied behaviour

Question 31

Valid positive and negative decisions

Answer 31

Positive - where the person is predicted to show the behaviour and shows it

Negative - where the person is not predicted to show the behaviour and does show it

Question 32

False positive and negative decisions

Answer 32

Positive - the prediction is that the person has the characteristic but does not

Negative - the prediction is that the person does not have the characteristic but does

Question 33

Base rate

Answer 33

the proportion of individuals in the population show show the behaviour of interest

Question 34

Selection ratio

Answer 34

the proportion of those assessed who can be allocated to the category of showing the behaviour