Chapter 4-reliability

Question 1

Classical Test Theory (CTT): Assumptions (4)

Answer 1

(1) Each person has a true score that would be obtained if there were no errors in measurement. Observed test score (X) = True test score (T) + Error (E)
(2) Measurement errors are random
(3) Measurement error is normally distributed
(4) Variance of OBSERVED scores = Variance of true scores + Error variance

Question 2

Reliable test

Answer 2

One we can trust to give us the same score for a person every time it is used.

Question 3

Can a measurement instrument be perfectly reliable?

Answer 3

No. No measurement instrument is perfectly reliable

Question 4

A person’s true score def

Answer 4

The hypothetical or ideal measure of a person’s attribute we aim to capture with a psychological test.
=> FREE FROM ERROR
Expected score over an INFINITE number of independent administrations of the test

Question 5

Independent administration def

Answer 5

Each time the test is taken is UNRELATED to PREVIOUS or FUTURE administrations
-> The person’s performance on one occasion doesn’t influence their performance on another.

Question 6

Mean error of measurement = ____
Errors are _______ with each other
True scores and errors are _______

Answer 6

0; UNcorrelated; UNcorrelated

Question 7

Two tests are parallel if: (3)

Answer 7

(1) EQUAL observed score MEANS
-> Comes from the assumption that True scores would be the same
(2) EQUAL ERROR VARIANCE
(3) SAME CORRELATIONS with other tests

Question 8

Random error characteristics (3)

Answer 8

(1) Random
(2) Cancels itself out
(3) Lowers reliability of the test

Question 9

Systematic error characteristic

Answer 9

Occurs when source of error always increases or decreases a true score
-> DOESN’T LOWER RELIABILITY of a test since the test is RELIABLY INACCURATE by the same amount each time

Question 10

Sources of Measurement Error (3)

Answer 10

(1) CONTENT Sampling Error
(2) TIME Sampling Error
(3) Other Sources of Error (e.g. observer differences)

Question 11

Content Sampling Error characteristics (3)

Answer 11

(1) Results from differences between the SAMPLE of items (i.e., the test) and the DOMAIN of items (i.e., all the possible items)
(2) When test items may not be representative of the domain from which they are drawn
(3) Low when test items are representative of the domain

Question 12

Time Sampling Error characteristics (2)

Answer 12

(1) Results by the choice of a particular time to administer the test
(2) Random fluctuations in performance from one situation or time to another

Question 13

Other Sources of Error characteristics (2)

Answer 13

(1) Scoring or administrative error
E.g., Adding scores with one another
(2) Tests scored or graded by different scorers

Question 14

Reliability Coefficient

Answer 14

Proportion of OBSERVED test scores accounted for by variability in TRUE scores.
=> Ratio of the variance of the true scores on a test to the variance of the observed scores
=> Measure of the accuracy of a test or measuring instrument obtained by measuring the same individuals twice and computing the correlation of the two sets of measures

Question 15

Standard Error of Measurement (SEM) def

Answer 15

Indicates the amount of uncertainty/error expected in an individual’s observed test score.
=> Corresponds to the SD of the distribution of scores one would obtain by repeatedly testing a person.
=> SD of the distribution of random errors around the true score!!

Question 16

Standard Error of Measurement (SEM) allows us to quantify the _______.

Answer 16

Amount of variation in a person’s observed score that measurement error would most likely cause

Question 17

High Reliability = ___ SEM
Low Reliability = ___ SEM

Answer 17

High reliability = Low SEM
Low reliability = High SEM

Question 18

Confidence Interval (CI) def

Answer 18

Confidence interval (CI) is a range of scores that we feel confident will include the true score.

Question 19

CI is used to compare scores to avoid ______________.

Answer 19

over-emphasizing differences

Question 20

Reliability of test can be increased by _______.

Answer 20

adding items

Question 21

Spearman-Brown formula def

Answer 21

Predicts the effect of lengthening or shortening a test on reliability.

Question 22

Test reliability is usually estimated with what methods? (4)

Answer 22

(1) Test-retest
(2) Alternate (Parallel) Forms
(3) Internal consistency
(4) Interrater/Raters

Question 23

Test-Retest method

Answer 23

TIME SAMPLING.
Administer the same test to the same group of examinees on 2 different occasions
Correlate first set of scores with second.
-> Level of reliability should match the construct
-> Higher when construct being measured is expected to be STABLE than when construct expected to CHANGE

Question 24

Alternate (Parallel) Forms method

Answer 24

ITEM SAMPLING.
Evaluate the test across different FORMS of the test.
Construct two similar forms of a test; administering both forms to the same group of examinees within a very short period of time.
-> Correlate the 2 sets of scores
-> The correlation coefficient as an index of reliability of either one of the forms.

Question 25

Internal Consistency method

Answer 25

We examine how people perform on similar subsets of items.
-> Selected from the same form of the measure.
One Test administration: A single form of a test is administered only once to a group of examinees.
=> How consistently the examinees performed across items or subsets of items on this single test form.

Question 26

Internal Consistency Validity - If scores are consistent across items on the same test form, then (2)

Answer 26

(1) Items came from the SAME CONTENT DOMAIN and constructed the same way
(2) Performance would GENERALIZE to other items from the same content domain

Question 27

How High Should INTERNAL CONSISTENCY Coefficients Be? (*confond pas avec d’autres coeff)

Answer 27

Higher for “narrow” constructs
Lower for “broader constructs
-> Very high may indicate insufficient sampling in the domain
E.g. Medium internal consistency is bad for a narrow construct (panic disorder), but not so bad for a broad construct (Neuroticism)

Question 28

What’s the older approach used to estimate the internal consistency of a test?

Answer 28

Correlate scores based on first half or second half of items.
Correlate scores based on odd or even items.
=> If the items get progressively more difficult, then you might be better advised to use the odd-even system, whereby one subscore is obtained for the odd-numbered items in the test and another for the even-numbered items.

Question 29

What’s the contemporary approach used to estimate the internal consistency of a test?

Answer 29

CRONBACH’S ALPHA = AVERAGE OF ALL POSSIBLE SPLIT-HALF RELIABILITIES
Unaffected by how items are arranged in the test
Contemporary approach to estimate internal consistency.
-> Most general method of finding estimates of reliability through internal consistency.

Question 30

Kappa formula

Answer 30

Interrater Agreement
Proportion of the potential agreement following CORRECTION FOR CHANCE.

Question 31

Interrater reliability

Answer 31

Degree of agreement among independent observers who rate/assess the same phenomenon.
Two or more people rate/score the same tests

Question 32

Domain Sampling Model

Answer 32

Model that holds that the TRUE score of a characteristic is obtained when ALL of the ITEMS in the domain are used to capture it.
-> Considers the problems created by using a limited number of items to represent a larger and more complicated construct.
-> Conceptualizes reliability as the ratio of the variance of the observed score on the SHORTER test and the variance of the LONG-run true score.

Question 33

Difficulties with applications of IRT

Answer 33

Requires a bank of items that have been systematically evaluated for level of difficulty.
-> Considerable effort must go into test development, and complex computer software is required.

Question 34

Test-Retest Method: Problems

Answer 34

CARRYOVER EFFECTS: Occurs when the first testing session influences scores from the second session.

Question 35

When there are carryover effects, the test-retest correlation usually ________ the true reliability.

Answer 35

OVERESTIMATES
-> This can happen because the participant REMEMBERS items or patterns from the first test, so their performance on the second test is less independent than it should be.

Question 36

In cases where the changes are ___, carryover effects do NOT harm the reliability.

Answer 36

SYSTEMATIC
E.g. When everyone’s score improves exactly 5 points. In this case, no new variability occurs.

Question 37

Important type of carryover effect

Answer 37

PRACTICE effects: improvement with practice
-> Because of these problems, the time interval between testing sessions must be selected and evaluated carefully.

Question 38

Because of practice effects, ______________ must be selected and evaluated carefully.

Answer 38

the time interval between testing sessions

Question 39

What method provides one of the most rigorous assessments of reliability commonly in use?

Answer 39

Parallel Forms Method
-> However: Test developers find it burdensome to develop two forms of the same test, and practical constraints make it difficult to retest the same group of individuals.
-> Many test developers prefer to base their estimate of reliability on a single form of a test.

Question 40

Problems with Split-Half method (2)

Answer 40

(1) The two halves may have different variances.
(2) The split-half method also requires that each half be scored separately, possibly creating additional work.

Question 41

What technique avoids the problems of split-half method and how?

Answer 41

The Kuder-Richardson technique avoids these problems because it simultaneously considers all possible ways of splitting the items. (Or Cronbach’s alpha)

Question 42

KR20 Formula

Answer 42

Simultaneously considers all possible ways of splitting the items. When test is dichotomous (Right or Wrong)

Question 43

Factor Analysis

Answer 43

Factor analysis is one popular method for dealing with the situation in which a test apparently measures several different characteristics.
-> Used to divide the items into subgroups, each internally consistent;
E.g. test that has submeasures for several different traits.

Question 44

Sources of measurement error: (3)

Answer 44

(1) Time sampling: The same test given at different points in time may produce different scores, even if given to the same test takers.
(2) Item sampling: The same construct or attribute may be assessed using a wide pool of items.
(3) When different observers record the same behavior: Different judges observing the same event may record different numbers.

Question 45

How do we assess measurement error associated with time sampling?

Answer 45

Test-retest method (coeff of stability)

Question 46

How do we assess measurement error associated with item sampling?

Answer 46

Parallel forms reliability

Question 47

How do we assess measurement error associated with “diff people judge same behavior”?

Answer 47

Adjusted index of agreement such as the kappa statistic.

Question 48

What to Do about Low Reliability? (3)

Answer 48

(1) Increase the # of Items
(2) Throw out items that run down the reliability (by running a factor/discriminability analysis)
(3) Estimate what the true correlation would have been (CORRECTION FOR ATTENUATION)

Question 49

How to interpret the Kappa stat?

Answer 49

Kappa = 0 is considered poor -> means the agreement is basically by chance.
Kappa = 1 represents perfect, complete agreement.

Question 50

When random error is HIGH on both tests, the correlation between the scores will be _____ compared to when the random error is ___.

Answer 50

lower; small

Question 51

Formulas for diff methods of reliability:
- Test-retest (time sampling error)
- Alternate forms (item sampling error)
- Internal consistency (item sampling)
- Interrater (observer difference)

Answer 51

Test-retest = Pearson’s (coeff stability)
Alternate forms = Pearson’s (coeff equivalence)
Internal consistency = Spearman- Brown formula, 𝐾𝑅20 formula, or Cronbach alpha
Interrater = Kappa formula

Question 52

Analysis for diff methods of reliability (3)

Answer 52

(1) Test-retest = Scores on first and second administration correlated
(2) Alternate forms = Scores on both tests are correlated
(3) Internal consistency = Scores on both halves are correlated OR Average correlation on all split halves
Interrater = Scores by both observers are correlated

Question 53

Difference Score def

Answer 53

Subtracting one test score from another
-> two different attributes

Question 54

Why are difference score unreliable?

Answer 54

Difference scores are unreliable because the random error from both scores is compounded and the true score is cancelled out.