L11 - Psych Assessment: Reliability

Question 1

What’s the difference between psychological assessment and psychological testing?

Answer 1

Psychological Assessment is the process of gathering all relevant psychological data about a person and interpreting those data in the context of the person’s broader well-being, and the benefit of society. BROADER

Psychological Testing is the process of administering one or more psychological tests. However, since both Testing and Assessment should be conducted in an ethical framework, there should be little difference in practice, that is, testing should only be conducted in a broader ethical and professional standards framework.

Question 2

Assumptions underlying the use of psychological tests?

Answer 2

1. Psychological traits and states exist.
2. Psychological traits and states can be quantified and measured.
3. Test-related behaviour predicts non-test related behaviour.
4. Tests and related techniques have strengths and weaknesses.
5. Various sources of error or unreliability are part of the assessment.
6. Assessment can be conducted in a fair and unbiased manner.
7. Psychological assessment benefits society.

Question 3

What should all psychological tests posses?

Answer 3

1. Established Validity.
2. Standardised administration and scoring
   - Clear rules for administration and scoring.
3. Adequate normative sample
4. Known reliabilities and standard errors of measurement.
5. Test publications and manuals.
6. Ongoing test development & revision.

Question 4

What is reliability?

Answer 4

The consistency of test scores – the study of relationships between items on the test on one or more testing occasions is usually reported as “reliability”. Study of reliability provides the key to interval estimation for one person’s score on a test. It is the extent to which a test correlated with itself in another form (version) or on another occasion. It is a necessary but not sufficient condition for validity, and sometimes termed as “internal validity”.

Question 5

What is Validity?

Answer 5

Degree to which inferences made on the basis of test scores are scientifically justified.

Question 6

What is variance?

Answer 6

Variance is the measure of the spread of individual differences on a test.
For two test with equivalent reliability and both measuring the same construct, the test with more variance (a larger S.D.) is more informative than a test with less variance.

Question 7

What is error variance?

Answer 7

Measurement error

the part of variance of scores that we don’t understand - isn’t conceptually theorised.

Question 8

What is the general model of reliability in classical test theory?

Answer 8

X = T (true score) + E (measurement error)

any observed score (X) has 2 components.

Question 9

Assumptions involved in general model of reliability?

Answer 9

Mean error of measurement = 0

True scores and errors are not correlated: rte=0

Errors on different measures are not correlated: re1e2= 0

• even in situations of unreliable measurement, average true score effects may emerge over multiple data collections / replications

Question 10

What can the variance of an observed score be deconstructed into?

Answer 10

σ2(X) = σ2(tryue) + σ2(error)

Question 11

What is the theoretical reliability coefficient?

Answer 11

r(tt) = true σ^2 / observed σ^2

this is the proportion of variance in observed scores that is due to variance in true scores.

a highly reliable test will have little error in its observed score variance.

Question 12

What is a theoretical entity?

Answer 12

True σ^2

hard to measure in real life..

Question 13

What do we do since true variance is actually a theoretical entity and is hard to measure?

Answer 13

We use a correlation coefficient instead, by using re-test correlations.

If r_tt = .5 then 50% of the variance in observed scores is due to the variance in true scores.

Question 14

What are some methods for estimating reliability?

Answer 14

➢ Test-retest reliability!!!!!
➢ Internal consistency (KR20 or Alpha)
➢ Split-half (correlation coefficient)
➢ Parallel-form or alternate-form (correlation coefficient)
➢ Inter-rater or inter-scorer reliability (correlation coefficient or Kappa).

Question 15

What is the standard error?

Answer 15

Term used to describe the standard deviation of a distribution for an inferred statistic, e.g. sample mean (standard error of the mean), or individual test score (standard error of measurement).

Question 16

How do we calculate standard error of measurement?

Answer 16

SEM or SEm or σmea = SDt√(1-rtt) = σx√(1-rxx)

But it really doesn’t tell us that much, just that the score will fall somewhere ‘human’

Question 17

What is predicted true score (PTS)?

Answer 17

PTS is the theoretically correct score around which to place confidence intervals for an individual test score. It is the most important to consider when rtt is relatively row (

Question 18

What is are confidence intervals centred around?

Answer 18

They are centred on the predicted true score.

Question 19

how do we calculate PTS?

Answer 19

PTS = ( rtt x Observed Score ) + ( Scale Mean x (1 - rtt))

Question 20

What are the three standard errors of measurement?

Answer 20

Standard error of measurement

Standard error of estimation: SEe = σx√[rtt(1-rtt)]

Standard error of prediction: SEp = σx√(1-rtt2)

Question 21

What is the standard error of estimation

Answer 21

This is used to establish confidence intervals around the PTS.

so then we can determine the likely location of an inidividuals true score… and compare with known population parameteres

Predicted True Score ± 1.96 x SEe

Question 22

What is Standard error of prediction?

Answer 22

This is used to observe changes in one person’s score over time.

Also used to make CI’s.

Predicted True Score ± 1.96 x SEp