WK 2 Norms and Reliability

Question 1

Describe the main premise of classical test theory and how it relates to reliability?

Answer 1

CTT says that every person’s observed score is made up of the true score (of the trait) as well as partly error. For a population, the total variance is the true variance + the error variance. Reliability refers to the proportion of true variance divided by the total variance. That is, reliability is directly influenced by true variance, but note - we can only ever estimate the true variance.

Question 2

Describe measurement error?

Answer 2

Measurement error is also known as error variance. It is made up of both systematic error (predictable and constant) and random error (unpredictable, unrelated, noise). Random error is good because it should balance out in the end and result in a similar mean. Systematic is less good, but if you know what could be affecting it, you can adjust your numbers accordingly.

Question 3

List common sources of measurement error

Answer 3

1. Test Construction
2. Test Administration
3. Test Scoring and Interpretation
4. Sampling Error
5. Methodological Errors

Question 4

Describe Test Construction error

Answer 4

Variation due to differences in items on same test or between tests

Question 5

Describe Test Administration error

Answer 5

Variation due to testing environment
(test-taker: anxiety, stress, drugs, sleep, physical discomfort)
(Examiner: appearance, demeanour)

Question 6

Describe Test Scoring and Interpretation error

Answer 6

Variation due to scoring and interpretation e.g. scoring a video on warmth behaviours of a mother towards aggressive child

Question 7

Describe Sampling Error

Answer 7

Variation due to representativeness of sample e.g. doesn’t gather sample that represents a population, instead only educated people

Question 8

Describe Methodological Errors

Answer 8

Variation due to poor training, unstandardised administration, unclear questions, biased questions

Question 9

What is the difference between CCT and IRT?

Answer 9

CTT assumes just two components to measurement and that all items have equal ability to measure the target in question.
IRT is very powerful in understanding the power of an item in finding latent traits, it examines items specifically and can reveal different levels of the latent trait being exmained

Question 10

IRT incorporates considerations of item difficulty and discrimination. Can you describe what they mean in the context of IRT?

Answer 10

Difficulty relates to the ability of an item to be completed, solved or comprehended
Discrimination refers to the degree to which an item differentiates between high and low levels of the construct. E.g. if the discrimination slope is steep, it is good at discriminating between different levels

Question 11

List the common estimates of reliability.

Answer 11

1. Test-retest reliability
2. Parallel and Alternate Forms Reliability
3. Internal consistency reliability (split-half, inter item correlation, Cronbach’s alpha)
4. Inter-rater/ inter-scorer reliability

Question 12

Describe Test-retest reliability

Answer 12

Estimate of reliability over time/ the consistency of a test over time
How? Correlate Pairs of scores from the same people, on the same test, at different time points
Good for? Stable variables e.g. Personality
Bad? Estimates tend to decrease as time passes
Not good for fluctuating variables e.g. Mood

Question 13

Describe Parallel and Alternate Forms Reliability

Answer 13

if the MEANS and VARIANCE are equal in both versions of a test = PARALLEL
If not = ALTERNATE
How? Correlate the scores of the same people measured by the different forms
E.g. Does cognitive function improve over time: use the Montreal Cognitive Assessment (MOCA): two different versions: Patient can’t use answers from first version to help them in second

Question 14

Describe split-half (internal consistency)

Answer 14

How? Correlate equivalent halves of the one test with each other, then generalise the half-test reliability to the full-test internal consistency reliability Spearman-Brown Formula By changing the ‘n’ of your final test, you can manipulate the reliability of your test.

S-B predicted reliability = (nhalf-correlation)/ 1 + (n-1) half-correlation

Question 15

Describe inter-item consistency/ correlation (internal consistency)

Answer 15

the degree of relatedness of items on a test. HOMOGENEITY. Basically you get the average of inter-item correlations

Question 16

Describe Kuder-Richardson Formula 20 (internal consistency)

Answer 16

Statistic of choice for determining the inter-item consistency of dichotomous (binary) items. I.e. yes/ no

Question 17

Describe coefficient/ Cronbach’s Alpha (internal consistency).

Answer 17

You get the mean of all possible split-half correlations, corrected by S-B formula, very popular approach for internal consistency. Values 0-1

Question 18

Describe Inter-rater/ inter-scorer reliability

Answer 18

Degree of agreement/ consistency between two or more scorers> correlate scores from different raters. Often used in behavioural measures. Aims to guard against biases or idiosyncrasies in scoring. Obtained by correlating scroes form different raters (INTRACLASS correlation for CONTINUOUS measures) allows you to adjust for systematic differences. (COHEN’S KAPPA for CATEGORICAL measures)

Question 19

What should you consider when choosing reliability estimates?

Answer 19

1. Homogenous/ Hetero
2. Dynamic/ static (over time how will it change)
3. Restricted/ not (restriction of or not enough restriction of range affects ur correlation)
4. speed/ power test (speed likely to be homogenous, power, likely to be heterogenous)
5. criterion-/ not criterion-referenced

Question 20

Why might we want to consider reliability of a single test score?

Answer 20

For example in the clinical setting, we want to know is one person’s score taking our test. We can use our reliability coefficient and generalise to a single score.

Question 21

How do you use reliability of tests to get precision?

Answer 21

1. Standard Error of Measurement (SEM)
   - estimates the single observed score and how close it is to the true score i.e. its’ precision/ amount of error
   - generally, higher reliability, lower SEM
   - Estimate the extent of deviation between observed and true score
   SEM CHANGES BASED ON SD and RELIABILITY OF TEST
2. Standard Error of the Difference (SED)
   - estimates the difference b/w 2 test scores and is it statistically significant
   - MUST use standardised variables, i.e. compare the apples with the apples, or convert the oranges to apples

Question 22

Explain the difference between norm-referenced and criterion-referenced tests?

Answer 22

Norm-referenced compares a single person’s test score to a normative sample. e.g. IQ test. Criterion-referenced tests compare a single person’s test score to a pre-determined standard criterion/ threshold. e.g. passing first aid course/ driving test

Question 23

What does Standardisation in sampling to develop norms mean?

Answer 23

It means the process of administering test to representative sample to establish norms

Question 24

What does SAMPLING in sampling to develop norms mean?

Answer 24

the selection of an intended population for the test, that has at least one observable characteristic

Question 25

What does STRATIFIED SAMPLING in sampling to develop norms mean? How is it different to STRATIFIED RANDOM SAMPLING?

Answer 25

Stratified purposefully includes a representation of subgroups in a population, Stratified Random Sampling forcibly samples in a way that makes the demographics equal e.g. equally from metropolitan and rural compared to natural higher metro, and natural lower numbers of rural

Question 26

What are the two less robust types of sampling, but also more common?

Answer 26

Purposive sample (arbitrarily selecting sample believed to be representative of the population
Incidental/ convenience sampling (convenient sample which may or may not be representative of population) > need to be cautious with generalisations of these findings

Question 27

What is the process of developing norms?

Answer 27

1. Administer the test with standard set of instructions
2. Recommend a setting for testing
3. Collect and analyse data
4. Summarise data using descriptive statistics including MEASURE OF CENTRAL TENDENCY and VARIABILITY
5. Provide a detailed description of standardisation and administration protocol, so others can replicate if they read ur paper in the same context

Question 28

What are the different types of norms? Describe them.

Answer 28

1. Percentiles NB: tend to be larger differences at edges e.g. big difference between ATAR 97 and ATAR 98, but one point difference, but smaller differences in middle but exaggerated point differences
2. Age norms
3. Grade norms
4. National norms
5. National anchor norms
6. Subgroup norms
7. Locla Norms

Question 29

What is a standard score?

Answer 29

A raw score converted from one scale to another that has a predefined scale i.e. a set mean and SD
e.g.
z-score (M=0, SD = )
t-score (M= 50, SD = 10)
You can calculate difference scores depending on set mean + SD, it all corresponds. Just needs to be pre-determined, can compare as long as everyone is on the same scale.

Question 30

Why is it important for test users to research all available norms before prescribing to a patient?

Answer 30

It is important to conduct culturally-informed assessment. E.g. no point comparing a low education to a university level education -> it affects the test results and misleads you