Brainscape's Knowledge GenomeTM

Browse over 1 million classes created by top students, professors, publishers, and experts.


See full index

PSYC30013 - Theme 4: Psychological Assessment > W11 - Reliability > Flashcards

W11 - Reliability Flashcards

1
Q

What is the difference between psychological research and psychological assessment

A

Psychological Research:

Generalisations about representative samples of people.

Psychological Assessment:

Generalisations about specific individuals (n = 1)

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
2
Q

What are some psychological assessment standards

A
  1. Nature of underlying construct(s)
  2. Basic psychometric principles and procedures: Requirements and Limitations
  3. Directions for administration and properties: Purpose, relevant standard errors, reliabality and validity
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
3
Q

What is a valid test

A

A test is valid if it accurately measures what it purports to measure

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
4
Q

What is a reliable test

A

Property of consistency in measurement.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
5
Q

Reliability and Validity. Necessary and Sufficiency.

A

Reliability is a necessary, but insufficient, requirement for validity.

(i.e. valid test cannot be unreliable, but reliable test may not valid)

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
6
Q

Is reliability a binary decision?

A

No. Reliability is continous, not categorical (Reliable/Not Reliable)

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
7
Q

What is the first equation of Classical Test Theory

A

Xi = T + Ei

Xi = Observed score on test occassion i

T = True Score

Ei = Error on test occassion i, Unsystematic variance.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
8
Q

What are the two properties of errors in classical test theory

A
  1. Endogenous: Factors about test-taker (client’s condition)
  2. Exogenous: Factors outside test-taker (psychologist measurement)
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
9
Q

What are the assumptions of Classical Test Theory

A
  1. Expected value of error = 0
  2. Errors do not correlate with one another
  3. Errors do not correlate with true scores
  4. Expected value of test = True Score (On repeated administration of test, on average people will score their true score)
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
10
Q

Elaborate on first assumption of classical test theory

A
  1. Expected value of error is zero

When all the errors on different test occassion adds up, it will be equal to 0

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
11
Q

Elaborate on the second assumption of classical test theory

A
  1. Errors do not correlate with one another

Errors on Testi does not affect error on Testj

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
12
Q

Elaborate on the third assumption of classical test theory

A
  1. Errors do not correlate with true score

r (te) = 0. Positive/Negative error is unrelated to true score

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
13
Q

Elaborate on the fourth assumption of classical test theory

A
  1. Expected value of test equals to true score

Average of all observed scores = True Score

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
14
Q

What is the second equation of Classical Test Theory

A

𝜎2x = 𝜎2t + 𝜎2𝜖 + 2cov(t,𝜖)

𝜎2x : Variance of observed scores

𝜎2t : Variance of true scores

𝜎2𝜖: Variance of error scores

2𝐶𝑜𝑣(𝜏,𝜖): Covariance between true scores and error scores, which is 0 under assumption (3)

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
15
Q

What is the third equation of classical test theory, relating to how reliability is calculated

A
  • 𝑅𝑒𝑙𝑖𝑎𝑏𝑖𝑙𝑖𝑡𝑦
  • 𝜌2𝑥𝜏
  • (𝜎𝜏2/𝜎𝑥2)
  • (𝜎𝜏2)/ (𝜎𝜏2+𝜎𝜖2)
  • (𝑆𝑖𝑔𝑛𝑎𝑙)/(𝑆𝑖𝑔𝑛𝑎𝑙+𝑁𝑜𝑖𝑠𝑒)

𝜌2𝑥𝜏: Theoretical reliability coefficient

𝜎𝜏2 : Variance True Scores

𝜎𝑥2 : Varaince Observed Scores

𝜎𝜖2 : Variance Error Scores

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
16
Q

Persepctives on reliability: Conceptual and statistical basis

A

Conceptual: Observed score in relation to:

(a) True Score; (b) Measurement error

Statistical; (a) Proportion of variance (b) Correlations

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
17
Q

Perspective 1: Using true score and proportion of variance

A

Ratio of true score variance to observed score variance (Same as second equation)

rxx = St2/ Sx2

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
18
Q

Perspective 2: Using measurement error and proportion of variance

A

Reliability is the lack of error variance

rxx = 1 - (S2𝜖/ S2x)

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
19
Q

Perspective 3: Using true score and correlations

A

Reliability is the squared correlation between
observed scores and true scores

rxx = r2xt

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
20
Q

Perspective 4: Using measurement error and correlation

A

Reliability is the lack of correlation between
observed scores and error scores

rxx = 1 - r2x𝜖

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
21
Q

Reliability in practice, since we do not know the true score varaince, what are some assumptions we add on in a parallel test.

A

We run parallel test: This test must have:

(1) Tau equivalent. True score on both test is the same
(2) Same level of error variance

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
22
Q

What are the 3 ways of testing reliability

A
  1. ) Test-retest
  2. ) Parallel-form reliability
  3. ) Split-half reliability

All three assumes that they are parallel forms of the test

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
23
Q

What is test-retest reliablity

A

Correlation between original test and retest (Same test, different time)

24
Q

What is the use of test-retest reliability

A

Useful for stable traits, not useful for transient states

25
Q

What are the cons of test-retest relability

A
  • Carryover Effects (Smaller gap between test and re-test)
    • Googling Answers
    • Bored
    • Remember test items
      • True score may vary
  • Participants may fail to return (Bigger gap between test and re-test)

Trade off between paticipants failing to return and carry over effects

26
Q

What is parallel-form reliability

A

Correlation between two parallel forms of the test

27
Q

Parallel form reliability: What must be ensured?

A
  1. Parallel form must measure same set of true scores
  2. Parallel form must have equal vairance as original form
28
Q

What are the pros of parallel form reliablity

A

It can be used on the same day

29
Q

What are the cons of parallel form reliablity

A
  • Might not truly be parallel
    • Affects true score
  • Carryover effects
    • Even though there is no direct memory effects from orginal test, they might still learn stuff
30
Q

What is split-half reliability

A

Correlations between 2 sub-tests split from 1 test

31
Q

What is the pros of split-half reliability

A

Only one test. Easy

32
Q

What are the cons of split-half reliability

A
  • Might not truly be parallel
  • Deflation of reliability estimate as subtests have only half the items compared to main test
33
Q

What is cronbach’s alpha

A

Means of all possible split-half reliabilites, scaled up to a full test instead of a half-test

34
Q

Is cronbach’s alpha legit. Why?

A

Not really. Provides a conservative, lower-bound estimate for reliability and recent study suggest it’s of limited use.

35
Q

What does the reliability coefficient (rxx) fail to do?

A

It does not tell us in test score units how much measurement error is ‘typical’ as it is not expressed in test units.

36
Q

What is standard error of measurenet (SEm)

A

Average error score (i.e. SD of erorrs)

37
Q

What is the formula for SEm

A

SEm = sx [√(1 - rxx)]

sx = standard deviation of observed scores

rxx = reliability coefficient

38
Q

If the test is completely unreliable, what is the standard error of measurement

A

SEm = sx [√(1 - rxx)]

If rxx = 0

Hence, SEe = Sx

Standard error of measurement = Standard deviation of observed scores

39
Q

If the test is completely reliable, what is the standard error of measurement

A

SEm = sx [√(1 - rxx)]

Since rxx = 1,

Hence, SEe = 0

No standard errors of measurement

40
Q

What is the direction of association between reliability and SEm as a proportion of SD

A

Negative. As reliability increases, SEm decreases

41
Q

According to Nunnally, what is the reliablility needed?

A

Bare minumum = 0.9

Desirable = 0.95

42
Q

What is the equation to predict a client’s true scores

A

T_hat = (rxx)(x) + (1 - rxx)(𝜇T)

  • T_hat = predicted true score
  • rxx = reliability
  • x = observed score
  • 𝜇T = population mean for the test (e.g. IQ = 100)
43
Q

What if the predicted true score if the test was completely unreliable

A

T_hat = (rxx)(x) + (1 - rxx)(𝜇T)

If rxx = 0,

T_hat = 𝜇T (population mean)

44
Q

What if the predicted true score if the test was completely reliable

A

T_hat = (rxx)(x) + (1 - rxx)(𝜇T)

If rxx = 1,

T_hat = x (that is, the observed scores)

45
Q

What is the direction of association between reliability and predicted true scores (T_hat)

A

As reliability increases, T_hat moves closer to observed scores.

As reliability decreases, T_hat regress towards the population mean.

46
Q

What is true score confidence intervals built upon

A

Standard error of estimation

47
Q

What is the equation for standard error of estimation

A

SEe = Sx [√rxx(1 - rxx )]

  • Similar to standard error of measurement (SEm). but with an extra rxx.
  • Note: Sx = Standard Deviation of Test
48
Q

What defines the 95% CI for predicted true scores

A

Lower Bound: [T_hat - (1.96 x SEe)]

Upper Bound: [T_hat + (1.96 x SEe)]

49
Q

What are the correlations between measures compared to correlations between constucts. Why?

A
  • Observed correlations between two measures x and y is always lower than true correlation between underlying constructs
    • Because observed correlations is attenuated/reduced by measurement error
50
Q

What does the disattenuation formula aim to

A
  • Estimates the correlation if 2 constructs were not affected by measurement error
    • Corrects for the fact that measurement error attenuates the correlation between 2 constructs measured
51
Q

What is the maximum correlation between 2 measures x and y

A

Max rxy = √rxxryy

  • rxx
    • ​Reliability of test x
  • rxy
    • ​Reliability of test y
  • rxy
    • ​Observed correlation of underlying constructs x and y
52
Q

What is the disattenuation formula

A

r’xy = rxy / √rxxryy

  • r’xy
    • ​Correlation between 2 constructs without measurement error
  • rxy
    • ​Obseved correlation between 2 constructs
  • rxx
    • Reliability of construct x
  • ryy
    • ​Reliability of construct y
53
Q

How to increase the correlation between test scores and constructs

A
  • Increase the relationship between construct and test
    • Quality of Items
  • Remove inconsistency in test administration and interpretation
    • Reduce measurement error (Exogenous)
  • Increase no. of test items
    • Quantity of items
54
Q

What is the Spearman-Brown Prophecy Formula

A

r’xx = (nrxx) / [1 + (n-1)(rxx)]

  • r’xx
    • ​Reliability of expanded test
  • rxx
    • ​Reliability of original test
  • n
    • Expansion factor (e.g. n = 2 means double items)
55
Q

What is the relationship between expansion factor and reliability of expanded test. What is the caveat?

A
  • Negative acceleration
    • Which has practical benefit
  • New items must be as good. If not, reliability of expanded test (r’xx ) might be even wprse
56
Q

If we know what is our desired reliability, what is the forumla to work out the expansion factor

A

n = (r’xx)(1 - rxx) / (rxx)(1-r’xx)

r’xx = Reliability of expanded test

rxx = Reliability of original test

PSYC30013 - Theme 4: Psychological Assessment (2 decks)

Brainscape helps you reach your goals faster, through stronger study habits.
© 2024 Bold Learning Solutions. Terms and Conditions