Lecture 4: Essential of Reliability

Question 1

Reliability

Answer 1

• suggest trustworthiness
• quality of test that suggest they are sufficiently consistent and free from measurement error
• consistency and precision of the results of the measurement process

Question 2

Measurement error

Answer 2

any fluctuation in scores that results from factors related to the measurement process that are irrelevant to what is being measured
- reliable scores should be free of measurement error

Question 3

treu score

Answer 3

• hypothetical entities that would result from error-free measurement
• goal of reliability analysis: to estimate true scores

Question 4

Individual’s true score

Answer 4

the average score in a hypothetical distribution of scores that would be obtained if the individual took the same test an infinite number of times

Question 5

observed score

Answer 5

derived from tests (= scores that the individuals actually obtain)

Question 6

any observed score (X0) is made up of two components

Answer 6

• the true score component

- the error score component

Question 7

True score component (Xtrue)

Answer 7

is construed to be that portion of observed score that reflects whatever ability, trait, or characteristic the test assesses

Question 8

error score component (Xerror)

Answer 8

difference between the observed score and the true score

- any other factor that may enter into the observed score as a conseqeunce of the measurement process

Question 9

sample variance 
(true scores in group data)

Answer 9

the average amount of variability in a group of scores

Question 10

sample variance consist of (two components)

Answer 10

• a portion that is true variance

- a portion that us error variance

Question 11

True variance

Answer 11

differences among the scores of individuals within a group that reflect their standing or position in whatever characteristic the test assesses

Question 12

error variance

Answer 12

differences among test scores that reflect factors irrelevant to what the test assesses
- reliability scores increases as the error component decreases

Question 13

Reliability coefficient (reliability)

Answer 13

defined as the ration of true score variance to total test score variance
- if test score variance = true variance (reliability = 1)

Question 14

Two-step process (Evaluation of reliability)

Answer 14

1. What are possible sources of error?

2. What is the magnitude of those errors?

Question 15

The relativity of reliability

Answer 15

• tests cannot be reliable, test scores are reliable!!

- score might be unreliable (due to test taker, testing situation)

Question 16

3 sources of error which can enter the test score

Answer 16

• Context in which tésting takes place
• test taker
• specific characteristics of the test itself

Question 17

Random measurement error vs. systematic measurement error

Answer 17

• some of the errors can be minimized (due to proper testing practice etc.)
• other cannot be eliminated but may be detected by variozs types of checks built into the test

Question 18

Sources of error

Answer 18

1. Interscorer difference
2. Time sampling error
3. Content sampling error
4. Interim inconsistency
5. Interim inconsistency and content homogeneity
6. Time and content sampling error

Question 19

Interscorer difference

Answer 19

• errors entering into scores whenever the element of subjectivity influences scoring
• refers to the variations in scores that stem from differences in the subjective judgements of the scorers

Question 20

Scorer Reliability

Answer 20

• method for estimating error due to interscorer differences
• 2 independent scorers (two independent scores are generated)
• correlation between the set of scores
  (for metric variables)

Question 21

Time sampling error

Answer 21

variability in test scores as a function of the fact that they are obtained at one point in time rather than at another

Question 22

Concept of time sampling error

- hinges on two related notion

Answer 22

1. Construct/behavior is liable to fluctuate in time

2. Construct/behaviors chnage at different paces in time

Question 23

Test-retest reliability

Answer 23

• test is administered twice on two different occasions to one or more groups of individuals
• correlation between the scores obtained from the two administrations
  = test-retest reliability coefficient
• crucial: length of time interval!

Question 24

Content sampling error

Answer 24

term used to label the trait-irrelevant variability that can enter into test scores as a result for fortuitous factors related to the content of the specific items included in a test

Question 25

Content sampling error can be due to..

Answer 25

1. faulty test constructions | 2. specific content which favors some test takers

Question 26

alternate-form reliability

Answer 26

- intended to estimate the amount of error in test scores that is attributable to content sampling error - two or more forms of a test (different in speciifc content) need to be prepared and administered to the same group of subjects - scores are correlated (alternate-form reliability)

Question 27

Split-half reliability

Answer 27

- administering a test to a group of individuals and create two scores for each person by splitting the test into two halves - the scores of the two halves are then correlated (split-half reliability coefficient)

Question 28

Spearman-Brown (S-B) formula

Answer 28

- based on the notion of all things being equal, ascore based on a longer test will be closer to the true test score than one based on a shorter test - the formula estimates the effect

Question 29

Spearman Brown formula (does what?)

Answer 29

the formula estimates the effect: that lengthening a test by any amount, or shortening a test to any fraction of its original size, will have on the obtained coefficient

Question 30

Some solutions to the Problem of How to split a test in halves ...

Answer 30

1. odd even split, or two halves | 2. for speed tests: two-trial reliability

Question 31

Interim inconsistency

Answer 31

error in scores that results from fluctuations in items across an entire test (low correlations among test items)

Question 32

What is interim inconsistency due?

Answer 32

1. content sampling | 2. Content heterogeneity

Question 33

Content heterogeneity

Answer 33

inclusion of items or sets of items that tap content knowledge or psychological functions that differ from those tapped by other items in the same test (only when the test should be homogenous)

Question 34

Internal consistency measures

Answer 34

statistical procedures designed to assess the extent of inconsistency across test items (split-half reliability coefficients accomplish this to some extent) - formulas that take into account the interim correlation

Question 35

interim correlation

Answer 35

the correlation between performance on all the items within a test

Question 36

Kudar Richardson formular (KR20) and coefficient alpha (cronbachs alpha) - function of two factors

Answer 36

- number of items in the test | - the ratio of variability in test taker*s performance across all the items in the test to total test score variance

Question 37

most frquently used formulars to calculate interim consistency

Answer 37

``` Kudar Richardson formular (KR20) coefficient alpha (cronbachs alpha) ```

Question 38

Kudar- Richardson formula

Answer 38

Applied to test whose items are scored as right or wrong (dichotomous) - dependent on the interim variability within a test

Question 39

Coefficient alpha (cronbach's alpha)

Answer 39

used for tests whose items have multiple possible responses | - dependent on the interim variability within a test

Question 40

Hetero vs. Homogeneity (Used in reference to the composition of ...)

Answer 40

1. the behavior samples (items) of a test | 2. group of test takers

Question 41

Time sampling an content sampling error combined

Answer 41

both can be estimated in a combined fashion for tests which require stability and consistency as results

Question 42

Delayed Alternate-Form Reliability

Answer 42

these coefficients can be calculated when two or more alternate forms of the same test are administered on two different occasions - addtional source of error: practise effects!