L2: Classical test theory

Question 1

what is the central statistic of classical test theory?

definition & synonyms

Answer 1

summed item score (sum of the scores on the items)
synyonyms: sum score, test score, score on the test

Question 2

what is the central idea behind classical test theory?

Answer 2

• every test taker has a true score on a test, which is underlying the summed item score
• true score: score that you would get using a perfect measurement instrument
• observed score will generally not equal the true score due to measurement error

Question 3

define measurement error

Answer 3

• other influences that cause random noise in the observed score
• goal is to minimize this error to improve reliability

Question 4

what are the 2 core assumptions underlying classical test theory?

assumptions & what follows

Answer 4

1. observed scores are true scores plus measurement error: Xo = Xt + Xe
2. measurement error is random
   what follows:
   - mean of the error = 0, because a nonzero mean would make the measurement error systematic
   - correlation between true score and error = 0 (Rte=0) because the mean of error is 0
   - observed score variance = true scores variance + error variance (So^2 = St^2+ Se^2

Question 5

what are the 4 ways of thinking about reliability?

Answer 5

as a proportion of variance:
- ratio of true score variance to observed score variance
- lack of error variance (reliable tests have minimal error variance)
as shared variance:
- correlation between observed scores & true scores (reliability is the squared correlation between these 2)
- lack of correlation between observed scores & error scores (highly reliable test shows lil correlation between observed scores & error)

Question 6

how can you define reliability as a proportion of variance?

Answer 6

comes from So^2 = St^2 (signal) + Se^2 (noise) assumption

high reliability when most of So^2 is St^2
low reliability when most of So^2 is Se^2

reliability = signal / signal + noise = St^2 / St^2 + Se^2 = St^2/So^2
and
reliability = 1 - noise / signal + noise = 1 - ( Se^2 / (St^2 + Se^2)) = 1- Se^2/So^2

Question 7

how can you define reliability as shared variance?

Answer 7

low reliablity if Xt (true score) shares not a lot of variance with Xo (observed score)
high reliablity if Xt shares a lot of variance w Xo

reliability = correlation (Xo, Xt)^2 = Rot^2 aka the amount of variance shared by observed score and true score
and
reliability = 1- correlation (Xo, Xe)^2 = 1- Roe^2 aka 1 - the amount of variance shared by observed score and error score

Question 8

what are the 4 models to test reliability from most restrictive to least restrictive?

Answer 8

1. parallel test (most restrictive)
2. tau equivalent test
3. essentially tau equivalent test
4. congeneric test (least restrictive)

Question 9

what are the restrictions of parallel test?

Answer 9

restriction on Xt1 (first test’ true score): needs to = Xt2
restriction on Se1^2 and Se2^2: need to be equal to each other (the variances of the measurement errors)
implication:
- mean of Xt1 and Xt2 need to be equal
- variance of Xt1 and Xt2 need to be equal
- mean of Xo1 and Xo2 need to be equal CAN BE TESTED
- correlation between Xt1 and Xt2 = 1 (Rt1t2 = 1)
- reliability of test 1 = reliability of test 2, so also Rt1o1 = Rt2o2 (correlation between observed and true score)
- variance of observed scores on test 1 and test 2 are equal (So1^2 = So2^2) CAN BE TESTED

Question 10

what is the model of observed score of test 1 and 2 according to parallel test? and of the true score?

Answer 10

model observed score on test 1: Xo1 = Xt1 + Xe1 and Se1^2 = Se2^2
model observed score on test 2:
Xo2 = Xt1 + Xe2 and Se2^2 = Se1^2
model for the true score:
Xt2 = Xt1

Question 11

what are the 2 types of reliability based on the parallel test model?

Answer 11

• test restest reliability
• split halves reliability

Question 12

what are the restrictions on the tau equivalent test?

Answer 12

restriction on Xt1 (first test’ true score): needs to = Xt2
no restriction on measurement error variances
implications
- mean of Xt1 = mean of Xt2
- variance of Xt1 = variance of Xt2 (St1^2 = St2^2)
- mean of Xo1 = mean of Xo2 CAN BE TESTED
- correlation between true scores on test 1 and test 2 = 1 (Rt1t2 = 1)

Question 13

what type of reliability is based on essential tau equivalent test model?

Answer 13

cronbachs alpha

Question 14

what is the model of observed score of test 1 and 2 according to essential tau equivalent test model? and of the true score?

Answer 14

model for true score:
Xt2 = a + Xt1
model observed score of test 1: Xo1 = Xt1 + Xe1
model observed score of test 2: Xo2 = a + Xt1 + Xe2

Question 15

what is the model of observed score of test 1 and 2 according to tau equivalent test model? and of the true score?

Answer 15

model for true score: Xt2 = Xt1
model observed score of test 1: Xo1 = Xt1 + Xe1
model observed score of test 2: Xo2 = Xt1 + Xe2

Question 16

what are the restrictionson essentially tau equivalent test?

Answer 16

restriciton on Xt2: a + Xt1 (true scores on second test are equal to true scores on first + any number)
no restriction on measurement error variances
implications:
- mean of true scores are different
- variance of true scores are equal (St1^2 = St2^2)
- correlation between true scores on test 1 and test 2 is 1 (Rt1t2 = 1)

Question 17

what is the model of observed score of test 1 and 2 according to congeneric test model? and of the true score?

Answer 17

model for true score:
Xt2 = a + bXt1
model observed score test 1:
Xo1 = Xt1 + Xe1
model observed score test 2:
Xo2 = a + bXt1 + Xe2

Question 18

what are the restrictions on the congeneric test?

Answer 18

Xt2 = a + bXt1
no restriction on measurement error variances
implications:
- mean of true scores are different
- variances of true scores are different
- correlation between true scores on test 1 and 2 = 1 (Rt1t2 = 1)

Question 19

what reliability measure is based on congeneric model?

Answer 19

omega

Question 20

what are 3 methods of estimating reliability?

Answer 20

1. alternate forms reliability
2. test retest reliability
3. internal consistency reliability

Question 21

what is the alternate forms reliability estimation technique?

Answer 21

• assumes parallel test model (meaning they measure same trait w same amoutn of error variance)
• apply 2 versions of the same test
• correlation between the 2 forms is the reliability

Question 22

what are the main challenges with the alternate forms reliability estimation technique?

Answer 22

• constructing the alternate forms of the same test is hard
• carry over effects (lack of motivation, fatigue etc)

Question 23

what is the test retest reliability estimation technique?

Answer 23

• assumes parallel test model
• apply same test twice to same group but at different times
• correlation is the reliability
• assumes the trait being measured remains stable over time (which isnt always the case)

Question 24

what are the main challenges with test retest reliability technique?

Answer 24

• carry over effects
• change in the true score: for constructs that fluctuate, like mood, the true score might change between the 2 tests

Question 25

what is the internal consistency reliability estimation technique?

Answer 25

- looks at how consistent the items within a single test are w each other, if items on a test measure the same trait, the test should be internally consistent - assumes parallel or essential tau equivalent test model (theres multiple internal consistency techniques) - consider (blocks of) items as seperate tests - formula will give the reliability

Question 26

what are the main challenges with the internal consistency reliabiltiy technique?

Answer 26

carry over effects (by end of test might be way better at answering or more tired or whatever so different parts of test might not be as consistent w each other)

Question 27

what are the 3 types of internal consistency reliability techniques?

Answer 27

1. split half - assumes parallel test model - split test in 2 parts - formula gives reliabilty (see table 6.2) 2. cronbachs alpha (or KR20 for binary items or standardized alpha) - assumes essential tau equivalent test model - each item considered separate part - formula gives reliability (see table 6.2) 3. omega - assumes congeneric test model (or stricter) - not applied in practice yet - estimate true score variance using unidimensional factor analysis - reliability is true score variance / observed score variance

Question 28

evaluate the 4 main ways of estimating reliability: alternate forms, test retest, split half, cronbachs alpha

Answer 28

- alternate forms: hardly feasible in practice, only in specific situations - test retest: important to establish reliability for new test, in research only rarely used - split half: depends highly on split used (undesirable), still used frequently - cronbachs alpha/kr20: very pop in research due to its ease, assumption (essential tau equivalence) hardly met; its the lower bound of the reliability! so the actual reliability will be equal to or higher then cronbachs alpha

Question 29

what are the COTAN guideliens about reliability for psych tests?

Answer 29

1. tests used for high impact inferences at individual level (ex: personnel selection, diagnosis of learning disabilities etc) - good: 0.9 or larger - sufficient 0.8-0.9 - insufficient: smaller than 0.8 2. tests used for less impact inference at individual level (descriptive use ex: study/therapy progress, career choice tests etc) - good: 0.8 or larger - sufficient: 0.7-0.8 - insufficient: smaller than 0.7 3. tests used at group level (ex: customer/team satisfaction, student evaluations, comparing groups etc) - good: 0.7 or larger - sufficient: 0.6-0.7 - insufficient: smaller than 0.6

Question 30

what is item discrimination?

Answer 30

differences in the item scores reflect differences in the construct (indicates how good the items are) - item total correlation: correlation between item scores & sum scores (how well does an item predict the total score on the test?) - corrected item total correlation: correlation between item scores & rest scores (how well does an item predict the other item scores excluding the item u are looking at)

Question 31

what is the problem with the item-total correlation? what resolves this?

Answer 31

its biased upwards as you are correlating an item with itself (partly) -> this problem solved with corrected item total correlation as it exluces the item itself

Question 32

what are the factors affecting reliability?

Answer 32

1. test length 2. sample heterogeneity 3. the correlation between pretest and posttest scores

Question 33

how does a tests reliability get affected by test length?

Answer 33

lengthening a test will generally increase reliability

Question 34

what is the equation for a test' reliability if its length has been changed?

Answer 34

Rnew = n x Roriginal / 1+(n-1)Roriginal where n = new amount of items / original amoutn of items

Question 35

how does a test reliability get affected by sample heterogeneity?

Answer 35

in homogeneous samples, reliability will be smaller than in heterogenous samples R = St^2 / St^2 + Se^2 in homogeneous samples, St^2 will be smaller as ppl are relatively similar to each other while in heterogenous samples St^2 will be larger as ppl are relatively dissimilar to each other

Question 36

how does a tests reliability get affected by the correlation between pretest and posttest scores?

Answer 36

difference score: Di = Xi (post test) - Yi (pretest): difference between post and pretest scores difference score reliabilty: 𝑅𝑑 = 𝑠𝑋𝑜^2 𝑅𝑋𝑋 + 𝑠𝑌𝑜^2 𝑅𝑌𝑌 − 2𝑟𝑋𝑜𝑌𝑜𝑠𝑋𝑜𝑠𝑌𝑜 / 𝑠𝑋𝑜^2 + 𝑠𝑌𝑜^2 − 2𝑟𝑋𝑜𝑌𝑜𝑠𝑋𝑜𝑠𝑌o pretest variance*reliability of pre tests + posttest variance*reliability of posttest - 2xcorrelation between pre and posttest*sd of pre test*sd post test important properties: - if correlation between pretest and posttest is large, reliability will be small - aka, difference reliability depends on reliability of the pretest and posttest - sensitive to difference in variance between Xi and Yi

Question 37

how can you estimate true scores?

Answer 37

1. true score estimate = summed item score 2. true score estimate: Xest = mean of Xo (observed scores) + Reliability (Xo (score of person youre interested in) - mean of Xo) - based on regression to the mean - due to unreliability, high scoring persons will likely score lower on a next test -> the lower the reliability, the more the true score estimate is pulled toward the mean

Question 38

what is the standard error aka standard error of measurment?

Answer 38

amount of error present in an individuals score Sem= So (sd of observed scores) * sqrt (1-Reliabilty) so higher reliability = smaller sem lower reliability = higher sem can be used to construct 95% confidence interval around true score estimate

Question 39

what is attenuation?

Answer 39

effect size/correlations observed will be SMALLER than the effect sizes / correlations of the true scores (because observed scores are diluted by error) + correlation is smaller & less likely to be significant for less reliable test anyway (so always consider reliabilty) aka when measurements arent reliable (cus theres measurement error), it weakens the relationships observed between variables

Question 40

what is wrong w corrections for attenuation?

Answer 40

the corrections done on the observed scores in order to remove the error in them, can be wrong !

Question 41

When is reliability high?

Answer 41

- when there is little error score variance relative to the true score variance - when the sum of the true score variance & error variance comes close to the true score variance - when the proportion of error variance in the observed variance is small

Question 42

what is the relationship between standard error of measurment & reliability?

Answer 42

A smaller standard error of measurement means that there is less deviation of observed scores from true scores, so a more reliable test

Question 43

Consider two tests that purport to measure the same construct. In a pilot study, a researcher finds their observed test score means to be the same, but their test score variances to not be the same. Which of the test models do these data follow?

Answer 43

tau equivalent test

Question 44

Which test model does Cronbach's alpha assume?

Answer 44

essentialy tau equivalent test model or stricter

Question 45

which model does the test-retest reliability assume?

Answer 45

parallel test model

Question 46

which model does the split half reliability assume?

Answer 46

parallel test model

Question 47

In a hypothetical dataset that contains the test scores on two tests, the true score mean and true score variance differ across the two tests. Which test model does this dataset follow?

Answer 47

the congeneric test model

Question 48

If you find the reliability of two tests measuring the same construct to be the same, what test model do these tests follow?

Answer 48

parallel test model

Question 49

In a hypothetical dataset that contains the test scores on two tests, the true score mean and true score variance are equal across the two tests. Which test model does this dataset follow?

Answer 49

parallel test model & tau equivalent test model

Question 50

Say you want to assess the consistency between the observed scores of one test and those of another test. Which method for estimating reliability do you use?

Answer 50

alternate forms reliability

Question 51

which criteria do 2 test forms need to meet, in order to legitimately use the alternate forms method of estimating reliability?

Answer 51

tests need to have identical true scores & identical error variance

Question 52

Jimmy conducts a study into aggression, for which he uses the Aggression Questionnaire (AGQ; Buss & Perry, 1992). He wants to know how reliable the AGQ is. Therefore, he lets his respondents fill in the questionnaire again. Which method of estimating reliability does Jimmy intend to use here?

Answer 52

test retest

Question 53

When someone calculates Cronbach’s alpha to estimate the reliability of a test, what general method of estimating reliability is that person using?

Answer 53

internal consistency

Question 54

For which of the reliability methods, it is problematic if the true scores differ across two tests?

Answer 54

- alternative forms - test retest

Question 55

for which reliability methods is it problematic if there are carry over effects?

Answer 55

- test retest - internal consistency - alternative forms

Question 56

What is a problem that arises from using the internal consistency method for estimating reliability?

Answer 56

A correlation between the item’s error scores caused by carry-over effects

Question 57

for which items are the the following reliability measure suitable? raw alpha, KR20, standardized alpha

Answer 57

raw alpha: LIkert scale items that do not differ in variance too much KR20: binary items standardized alpha: Likert scale items that differ substantially in their item variance

Question 58

what is a difference between raw alpha, KR20, and standardized alpha? not concerning which items they are used on

Answer 58

Raw alpha and KR20 are based on the item covariances and item variances; standardized alpha only uses item correlations

Question 59

In what situation is it good to use standardized alpha?

Answer 59

When the item variances differ a lot from each other and thus the test score mostly reflects items with high variances

Question 60

What can we do to improve the reliability of a test?

Answer 60

Add more items to the test that are perfectly parallel to the original items

Question 61

is the reliabiltiy of the test smaller in a heterogenous sample or homogeneous sample?

Answer 61

homogenous sample

Question 62

If the pretest and posttest are both reliable, the reliability of the difference scores can still be relatively small if the pretest and posttest are...

Answer 62

highly correlated

Question 63

where can you find the split halves reliability? what about the reliablity of one halve?

Answer 63

split halves reliabilty: spearman brown coefficient reliability of one halve: correlation between forms

Question 64

Which statistic do we use when we want to know the consistency between one item and the other items of a test?

Answer 64

Corrected item-total correlation

Question 65

define reliability

Answer 65

consistency or stability of test scores across repeated applications. It's a crucial aspect of psychometrics because it determines how much trust can be placed in test results.

Question 66

what is the main assumption of the parallel test?

Answer 66

that 2 tests measure the same trait w equal true scores and error variances

Question 67

what is the main assumption of the tau equivalent test?

Answer 67

that 2 tests have equal true score variances but can differ in error variances

Question 68

What is the main assumption of the congeneric tests?

Answer 68

that a linear relationship between true scores across 2 tests exists, allowing for flexibilty in error & true score variances

Question 69

what is the domain sampling theory?

Answer 69

treats test items as a sample from a larger domain (bucket) of possible items - the reliability of a test is the average correlation between all possible pairs of tests drawn from that domain: basically how consistent the test resuts would be if you made different tests by pulling out different sets of items from the bucket of all possible questions basically this theory is saying "we want to know if the questions we randomly picked give a reliable pic of the persons true ability, even if we swapped them out for different questions from the same big bucket"