Lecture 1: Classic Test Theory Flashcards

Question 1

Q

When psychologists assess the quality of a test, what two metrics do they typically refer to?

Answer

A

Validity and reliability

Question 2

Q

What is test variance and how do you calculate it? (2)

Answer

A

Item variance is the measure of dispersion of the scores on item i. The test variance is the measure of the dispersion of the test scores. A covariance matrix is constructed in which the variance of each item is along the diagonal and the covariance between each item is displayed. The test variance is the sum of all these values in the matrix or the variance of the final test scores, its the same value.

Question 3

Q

Whats the difference between covariance and correlation if there is one?

Answer

A

Covariance is an unscaled measure of association between variables, correlation is a scaled measure of association between variables between -1 and 1

Question 4

Q

What can be used to infer the dimensionality of a test in CTT?

Answer

A

Principle Component analysis (PCA)

Question 5

Q

What is meant by Principal component analysis (PCA)?

Answer

A

Principal component analysis (PCA) is a technique for reducing the dimensionality of such datasets, increasing interpretability but at the same time minimising information loss. It does so by creating new uncorrelated variables that successively maximise variance. E.g reducing something (e.g tumour) with 30 dimensions (smoothness, volume) to two principle components

Question 6

Q

Summarise the main steps of how PCA is calculated

Answer

A

We calculate the covariance matrix of our data, we calculate the eigenvectors of the covariance matrix, and this gives us our principal components. The eigenvector with the largest eigenvalue is the first principal component, and the eigenvector with the smallest eigenvalue is the last principal component.

Question 7

Q

What does Xgp represet in CTT?

Answer

A

𝑋𝑔𝑝 is a random variable denoting the repeatedly sampled measurements of test g on subject p.

Question 8

Q

What two fundamental equations can be derived from CTT?

Answer

A

𝐸 (𝑋𝑔𝑝) =𝜏𝑔𝑝
The expected value of Xgp is equal to the true value
𝐸𝑔𝑝 =𝑋𝑔𝑝 −𝜏𝑔𝑝 (for a fixed subject)
(error = observed score - true score)

Question 9

Q

What three assumptions are there within CTT?

Answer

A

(a) the measurement is on an interval scale;
(b) the variance of observed scores 𝜎2 𝑋𝑔 is finite;
(c) the measurements are repeatedly sampled in a linear, experimentally independent way.

Question 10

Q

What 8 properties are derived from CTT?

Answer

A

The expected error score is zero;
The correlation between true and error scores is zero;
The correlation between the error score on one measurement and the true score on another measurement is zero;
The correlation between errors on linearly experimentally independent measurements is zero;
The expected value of 𝑋𝑔𝑝 over persons is equal to the expected value of the true score random variable over persons;
The variance of 𝐸𝑔𝑝 over persons is equal to the expected value, over persons, of 𝜎2 𝑋𝑔𝑝 (variance within persons);
Sampling over persons with any 𝜏𝑔𝑝, the expected value of the error score random variable is zero;
The variance of observed scores is the sum of the variance of true scores and the variance of error scores;

Question 11

Q

Give proof that the expected error score is 0

*Not required but gives an idea of how CTT is derived

Answer

A

𝐸 (𝑋𝑔𝑝) =𝜏𝑔𝑝 (fundamental Eq. 1)
𝐸𝑔𝑝 =𝑋𝑔𝑝 −𝜏𝑔𝑝 (fundamental Eq. 2)

𝐸 (𝐸𝑔𝑝) = 𝐸 (𝑋𝑔𝑝 −𝜏𝑔𝑝) = 𝐸 (𝑋𝑔𝑝) −𝐸 (𝜏𝑔𝑝)* = 𝜏𝑔𝑝 - 𝜏𝑔𝑝 = 0

*For one person 𝜏𝑔𝑝 is fixed

Question 12

Q

Give the proof for the following:

The correlation between true and error scores is zero;
The correlation between the error score on one measurement and the true score on another measurement is zero;
The correlation between errors on linearly experimentally independent measurements is zero;

*Not needed to reproduce exact theorems

Answer

A

The correlation between true and error scores is zero;
𝑋𝑔 =𝑇𝑔 +𝐸𝑔
or 𝑋 =𝑇+𝐸
𝐸 𝐸𝑔𝑝 =0 (property 1) → 𝐸 (𝐸𝑔 |𝑇𝑔 =𝜏𝑔𝑝) =𝐸 (𝐸𝑔𝑝) =0 for all 𝜏𝑔𝑝 → 𝜌 (𝐸𝑔,𝑇𝑔 )=0

If you know the error is 0 for each person, the expected value of the error is also 0. Therefore there cannot be a correlation between the error and the true score.

The correlation between the error score on one measurement and the true score on another measurement is zero;
𝐸 𝐸𝑔 =0 (property 1) → 𝐸 (𝐸𝑔 |𝑇ℎ =𝜏ℎ𝑝) =0 for all 𝜏ℎ𝑝 → 𝜌 (𝐸𝑔,𝑇ℎ) = 0

Same logic; if error is zero, it cannot be correlated with true score

The correlation between errors on linearly experimentally independent measurements is zero;
𝐸 (𝐸𝑔) =0 (property 1) → 𝐸 (𝐸𝑔 |𝐸ℎ =𝐸ℎ𝑝) =0 for all 𝐸ℎ𝑝 → 𝜌 (𝐸𝑔,𝐸ℎ) =0

Same logic; if error is zero, it cannot be correlated with other errors

Question 13

Q

Give the proof of property 8: The variance of observed scores is the sum of the variance of true scores and the variance of error scores;

Answer

A

𝜌 𝐸𝑔,𝑇𝑔 =0 (property 2)
𝑋𝑔 =𝑇𝑔 +𝐸𝑔 (population model)

𝜎2 (𝑋𝑔) = 𝜎2 (𝑇𝑔 +𝐸𝑔 )*= 𝜎2(𝑇𝑔)+𝜎2 (𝐸𝑔) +2𝜎( 𝑇𝑔,𝐸𝑔)
→𝜎2 𝑋𝑔 =𝜎2(𝑇𝑔)+𝜎2 𝐸𝑔
or 𝜎2/𝑋 =𝜎2/𝑇 +𝜎2/𝐸

*Covariance matrix

Question 14

Q

How can reliability be defined according to these terms (conceptually, and proof shown)

Answer

A

Conceptually: That it is the squared correlation between the test score and the true score for a participant

Using fundamental Equations 1 and 2, and property 2, reliability can be defined as:

𝜌 (𝑋𝑔,𝑇𝑔) 
= 𝜎(𝑋𝑔,𝑇𝑔) / 𝜎 (𝑋𝑔) 𝜎(𝑇𝑔) 
= 𝜎(𝑇𝑔 +𝐸𝑔,𝑇𝑔) / 𝜎 (𝑋𝑔) 𝜎(𝑇𝑔) 
= 𝜎 (𝑇𝑔,𝑇𝑔) +𝜎(𝐸𝑔,𝑇𝑔) / 𝜎 (𝑋𝑔) 𝜎(𝑇𝑔) 
= 𝜎2 (𝑇𝑔) +0 /𝜎 (𝑋𝑔) 𝜎(𝑇𝑔)
= 𝜎 (𝑇𝑔)/ 𝜎 (𝑋𝑔)

→𝜌𝑋 =𝜌^2𝑋,𝑇 =
= 𝜎(𝑋𝑔,𝑇𝑔) ^2
= (𝜎(𝑇𝑔)/𝜎 (𝑋𝑔))^2 =
= 𝜎2 (𝑇𝑔)/𝜎2 (𝑋𝑔)

corr between test score and true score
= formula for corr
= Xg = Tg + Eg
=covar of T+E & T can be written as covar of T & T + covar of E & T (rule)
=covar of T = the var, covar between E and T is 0 as explained before
= the 𝜎 (𝑋𝑔) cancel, leaving one on top
=Not there yet, reliability of x = corr between x and t squared (how much, in %, var of the total score variance is due to the true score)
=the corr squared = what we derived before
= the var of t / the var of x

Question 15

Q

How insightful is this definition of reliability?

Answer

A

insightful as 𝜎^2 (𝑋𝑔) =𝜎^2(𝑇𝑔)+𝜎^2 (𝐸𝑔) (property 8)

Question 16

Q

These are theoretical equations, we cannot calculate them without the variance of true scores. How do we try to do this?

Answer

A

The concept of parallel tests: if you have test h with a parallel test form g.

Question 17

Q

What are the assumptions of parallel tests

Answer

A

You assume the true scores are identical on the two tests as well as the variance of the error.

Question 18

Q

How are parallel tests g and h defined mathematically?

Answer

A

𝜏ℎ𝑝 =𝜏𝑔𝑝 → 𝑇ℎ =𝑇𝑔 =𝑇
The true score on one test is the same as the true score of another for one subject

𝜎2(𝐸ℎ𝑝)=𝜎2(𝐸𝑔𝑝) →𝜎^2 (𝑋ℎ) =𝜎^2 (𝑋𝑔) =𝜎^2(𝑋)
If you have the same error variance, you have the same test score variance.

Question 19

Q

How do these definitions help us calculate the reliability of a test?

Answer

A

If you have the same true score between tests and the error and test score variance between tests then the correlation between the test scores is equal to the reliability of each test

Question 20

Q

Prove mathematically that the correlation between the test scores is equal to the reliability of each test

Answer

A

𝜌 (𝑋ℎ,𝑋𝑔)
= 𝜎 (𝑇ℎ +𝐸ℎ,𝑇𝑔 +𝐸𝑔)/ 𝜎 (𝑋ℎ) 𝜎 (𝑋𝑔)
= 𝜎 (𝑇ℎ,𝑇𝑔) +𝜎 (𝐸ℎ,𝑇𝑔) +𝜎 (𝐸𝑔,𝑇ℎ) +𝜎 (𝐸𝑔,𝐸ℎ) / 𝜎 (𝑋ℎ) 𝜎 (𝑋𝑔)
= 𝜎 (𝑇ℎ,𝑇𝑔 )+0+0+0 / 𝜎 (𝑋ℎ) 𝜎 (𝑋𝑔)

= plug in the X = T + E from CTT divided by std of X for each (formula for covar)
= Same trick T+E,T = T+T, T + E for both (property 3 + 4)
=covariance of E with anything = 0, corr of true scores/ std of test scores

since parallel tests also say:
𝑇ℎ =𝑇𝑔 =𝑇
𝜎2 𝑋ℎ =𝜎2 𝑋𝑔 = 𝜎2 𝑋 :

𝜎 (𝑇ℎ,𝑇𝑔)/𝜎 (𝑋ℎ) 𝜎 (𝑋𝑔)
= 𝜎^2 (𝑇) / 𝜎^2 𝑋
=𝜌^2𝑋,𝑇
=𝜌𝑋

Question 21

Q

What is the next step towards really calculating reliability?

Answer

A

Chronbachs alpha: Assumes all the items are really parallel tests/items to use their ideas to calculate reliability. This is the most used index for reliability in psychology.

Question 22

Q

How is Cronbach’s Alpha given mathematically ad conceptually?

Answer

A

𝛼𝜌𝑋 = (𝑛/𝑛−1)*(𝜎|2𝑋|−Σ𝜎𝑖^2 / 𝜎|2𝑋|)
= (𝑛/𝑛−1) (ΣΣ𝑖≠𝑗𝜎𝑖𝑗 / 𝜎|2𝑋| )
where n is the number of items

= 𝑛/𝑛−1 * test score variance - sum of item variances (diag of covar matrix) / test variance
= 𝑛/𝑛−1 * sum of all elements of covar matrix - diagonal of covar matrix / sum of all elements of covar matrix

so n/n-1 * sum of the undiagonal element for a covariance matrix divided by the sum of the matrix

Question 23

Q

When does Chronbach’s Alpha give the exact reliability? What consequences does this have?

Answer

A

Cronbach’s alpha gives the reliability if the test is essential tau-equivalent, i.e.,
𝑇𝑖 =𝑎𝑗𝑘 +𝑇𝑗 → 𝜎^2 (𝑇𝑖) =𝜎^2 (𝑇𝑗)

Since this is rarely ever fulfilled, Chronbach’s alpha underestimates the reliability. It is essentially the lower bound of the reliability

Question 24

Q

How is Cronbach’s alpha a lower bound?

Answer

A

When Cronbacks alpha is derived reliability can be rewritten as:
𝜌𝑋 =𝐴+ 𝛼𝜌−𝐵
reliability = A + Cronbachs alpha - B

→If the item/parts are essential tau-equivalent: 𝐴=𝐵 so that 𝐴−𝐵 =0
→If not, A will always be larger than B: 𝐴>𝐵. Thus, Cronbach’s alpha is a lower bound!

Note: A and B signify other parts of the derived equation that satisfy these points

Question 25

Q

How was Chronbachs Alpha first introduced and how is it relevant to how it is treated today?

Answer

A

Cronbach’s alpha was just one of 6 proposed measures of reliability in the same paper although it is sometimes treated as the only measure. It is 𝜆3 when 𝜆1-𝜆6 were proposed in that paper. Chronbach reinvented it but it existed before

Question 26

Q

What measure of reliability was later proposed by woodward and bentler (1980)?

Answer

A

Greatest lower bound:

Under Classical Test Theory, the variance of the test scores is given by:
𝜎^2 (𝑋) =𝜎^2(𝑇) +𝜎^2 (𝐸)

Then, the greatest lower bound (Woodward & Bentler, 1980) is given by:
𝐺𝐿𝐵𝜌𝑋 = 1− max(𝐸𝜎^2(𝐸𝑖)) / 𝜎2(𝑋)

The maximum variance of E possible, given that 𝜎2 𝑋 should
be positive

Question 27

Q

How can GLB be estimated?

Answer

A

GLB can only be estimated using an algorithm (Theres an R package)

Question 28

Q

How do 4 of these reliability estimates compare to each other in regards to size? What could be inferred from this?

Answer

A

𝜆1-6 were proposed, 𝜆3 is C.A

𝜆1 < 𝛼𝜌𝑋 ≤𝜆2 ≤ 𝐺𝐿𝐵𝜌𝑋
𝜆1 < 𝛼𝜌𝑋 =𝜆2 = 𝐺𝐿𝐵𝜌𝑋 (for essentially tau-equivalent items)

This could indicate that GLB would be the safest to use since in the worst case it is equal to CA, in the best case its larger than CA. This shows that CA is valuable because it follows from parallel testing but there are indices out there, some of which are arguably better.

Question 29

Q

Name 6 other practically useful statistics from CTT ad name when they are useful

Answer

A

• Split half reliability: 𝑆𝐵𝜌𝑋 = 2𝜌𝑋1𝑋2 / 1+𝜌𝑋1𝑋2
> 𝑋1 and 𝑋2 are the two halves
> If lower bounds are not meaningful, e.g., in randomized experimental trials

• Test-retest reliability: 𝑟𝑒𝑡𝑒𝑠𝑡𝜌𝑋 =𝜌𝑋1𝑋2
> 𝑋1 and 𝑋2 are the two administrations
> If the underlying construct is stable enough, and no memory effects

• Standard Error of Measurement (SEM): 𝑆𝐸𝑀 =𝜎𝐸 =𝜎𝑋(1−𝜌𝑋)
> To determine a confidence interval for 𝜏𝑝

• Correction for attenuation (reduction in strength of signal?): 𝜌𝑇𝑔𝑇ℎ = 𝜌𝑋𝑔𝑋ℎ / 𝜌𝑋𝑔𝜌𝑋ℎ
> Where 𝑋𝑔 is from one test and 𝑋ℎ from another

• Item mean
> As a measure of item difficulty

• Item-rest correlation
> As a measure of item discrimination

Question 30

Q

What criticisms have there been for test theory?

Answer

A

The true score is nothing more than the expected value of 𝑋on test 𝑔
𝑋𝑔𝑝 =𝜏𝑔𝑝 + 𝐸𝑔𝑝
𝑋𝑔𝑝 =𝐸(𝑋𝑔𝑝) + 𝐸𝑔𝑝

As 𝐸(𝑋𝑔𝑝) is just a statistical expectation about test 𝑔:

The true score does not necessarily correspond to a unidimensional construct score
Statistics from CTT depend on both the item properties and the properties of the subjects
The true score contains irrelevant -but systematic- item specific effects

Question 31

Q

What is meant by saying that the true score does not necessarily correspond to a unidimensional construct score?

Answer

A

The true score is just an expected value on a test, it is a statistical thing. Some people seem to give it some kind of magical status, they see it as the construct or as a dimension, latent variable but it is just an expected value. If your data contains a unidimensional construct or nicely/ accurately measures a unidimensional construct, then your true score might represent this but still you’re not sure.

Question 32

Q

What is meant by unidimensional constructs and why would we want to measure them?

Answer

A

Constructs with just one dimension e.g working memory, extraversion, openness to experience. As opposed to higher order constructs with multiple dimensions such as intelligence, emotional intelligence etc. The benefit of trying to measure unidimensional constructs is that you can infer that a measure of a score reflects a high level of one variable rather than a possible high score in a number of variables.

Question 33

Q

Every test has a true score. You can therefore apply CTT to any test. You can calculate sum score or reliability and you have applied classical test theory.

What is wrong with this?

Answer

A

You may not have checked if this is the right thing to do and if CTT is appropriate for your data. E.g a with a three unrelated question questionnaire you could likely get good test retest reliability, sum score etc despite the test measuring nothing.

Alternatively a test could be measuring two constructs. This may be observed by looking at the correlation matrix and seeing that there are two groups of questions correlating with each other. Cronbach’s Alpha and GLB, however, sum all items and so how do you interpret this sum score since it has two dimensions?

Question 34

Q

What does it mean to say that “Statistics from CTT depend on both the item properties and the properties of the subjects”?

Answer

A

Each intelligence test (X𝑔,Xℎ,X𝑓,etc.) contains a different true score (𝜏𝑔,𝜏ℎ,𝜏𝑓, etc.) with its own scale depending on

The number of items (10 items vs 1000 items means a difference of steps of 0.1 or 0.001 on your scale)
The difficulty/discrimination of the items
The skill of the subjects that took the test (you in smart vs dumb sample)

However all of these tests would have the same true score since they’re supposed to be measuring the same thing.

As a result, all statistics from classical test theory depend on
1. the properties of the test
• Item difficulty and item discrimination
2. the properties of the sample
• Mean and variance of the true scores of the subjects

Question 35

Q

How does variance affect reliability?

Answer

A

Say a group was measured on a construct that they do not differ much in accurately (e.g uva professors and intelligence) then it will be hard to score a high reliability even if you always get similar answers since variance is factored into the reliability equation

Question 36

Q

What does it mean to say that the true score contains irrelevant -but systematic- item specific effects?

Answer

A

E.g in the following example

At parties, I always talk to everybody
I like giving talks for large audiences
In business meetings, I am the centre of attention
If someone hurts me, I will stand up for myself

All involve extraversion and this plays a factor in the decision made, however there is the extra noise (item specific error) in each item in addition to the measurement error variance. For example that each item takes place in a different setting. Perhaps someone is into parties and has no experience with business. Perhaps someone is very involved in their work and does not go to parties. The sum score calculates the answers to all these items as part of the true score, however, despite the extra error since these will also likely carry reliability.

Question 37

Q

What was proposed to deal with these criticisms?

Answer

A

Latent variable models

Question 38

Q

How do Latent Variable models deal with these criticisms?

Answer

A

They specify an explicit measurement model, a statistical model which describes the relationship between the construct and the item. This is different to CTT where the true score is not an explicit construct it is an expected value. In CTT the expected value of a score on item i is equal to the true score, in LVM the expected value depends on the latent variable.

Question 39

Q

What is meant by latent variables and item parameters?

Answer

A

Latent variable (person parameter):
Unobserved dimension of individual differences
that underlies all items in a test

Item parameters:
Model the item properties (comparable to e.g.,
item-rest correlation, item means)

Question 40

Q

Show mathematically how in four important equations in Latent Variable Models, the latent variables play a factor for measurement model 𝐸 (𝑋𝑝𝑖|𝜃𝑝)

Answer

A

𝜃𝑝 refers to latent variables/ person parameters
𝜐𝑖,𝜇1𝑖,𝛽𝑖, 𝜋1𝑖, +𝜆𝑖 refer to item parameters

• factor analysis:
e.g., 𝐸 (𝑋𝑝𝑖|𝜃𝑝) =𝜐𝑖 +𝜆𝑖𝜃𝑝

• Item response theory:
e.g., 𝐸 (𝑋𝑝|𝑖 )𝜃𝑝 = exp(𝛼𝑖𝜃𝑝+𝛽𝑖) / 1+exp(𝛼𝑖𝜃𝑝+𝛽𝑖)

• Latent class analysis: e.g., 𝐸(𝑋𝑝𝑖| 𝜃𝑝) =𝜋|𝜃𝑝 0𝑖|
 ×𝜋|1−𝜃𝑝 1𝑖|

• Latent profile analysis: e.g., 𝐸 (𝑋𝑝𝑖| 𝜃𝑝) =𝜇|𝜃𝑝 0𝑖|×𝜇|1−𝜃𝑝 1𝑖|

Question 41

Q

What does a structural model, 𝐸(𝜃𝑝|𝐵𝑝) represent?

Answer

A

Structural model, 𝐸(𝜃𝑝|𝐵𝑝):
A statistical model describing the relation between construct and other variables, 𝐵𝑝
• E.g., similar to a regression model, ANOVA, t-test, etc

Question 42

Q

What is the relationship between the structural model and the measurement model?

Answer

A

With the structural model you can take your latent variable of the construct and apply it to a regression model, anova etc. The measurement model accounts for all the measurement properties of the items so that you can safely make inferences with the structural model about the latent variable which do not suffer from problems which the true score suffers from

Question 43

Q

How do latent variable models address the following criticism of CTT?

The true score does not necessarily correspond to a construct score

Answer

A

Latent variable models are falsifiable
• There will be no latent variable in the data of unrelated questions
• only item specific effects which inflate test-retest reliability
• You will be able to tell from the latent variable variance (approaches 0)

A latent variable model with one latent variable will also not
fit the multidimensional data (model fit indices will indicate).
Instead, a latent variable model with two latent variables will fit these data (model fit indices will indicate)

Question 44

Q

How do latent variable models address the following criticism of CTT?

The true score depends on the scale of test g

Answer

A

In a latent variable model, test and sample properties are separated:
• Test properties will be captured by the item parameters
• Sample properties will be captured by the latent variable
Thus, all intelligence tests will be measuring the same latent variable

Question 45

Q

How do latent variable models address the following criticism of CTT?

The true score contains irrelevant but systematic
item specific effects

Answer

A

Recall that a latent variable is defined as : “Unobserved dimension of individual differences that underlies all items in a test”

Question 46

Q

Thus summarise the advantages of latent variable models and give two disadvantages

Answer

A

Latent variables explicitly model the dimensionality of a test

Latent variable models are falsifiable

Latent variable models are not test and sample dependent
Latent variable models explicitly account for item specific error

But:
Require much larger sample sizes
Statistically more complex