Reliability

Question 1

What is reliability in measurement?

Answer 1

Consistency in measurement.

Question 2

What does a reliability coefficient represent?

Answer 2

It indicates the ratio between true score variance and total variance.

Question 3

What does a reliability coefficient of 0 indicate?

Answer 3

No reliability.

Question 4

What does a reliability coefficient of 1 indicate?

Answer 4

Perfect reliability.

Question 5

What is the formula for observed score?

Answer 5

Observed score = true score + error (X = T + E).

Question 6

What does Cronbach’s Alpha indicate if it’s too high?

Answer 6

Redundancy in test items.

Question 7

What is measurement error?

Answer 7

Factors affecting the measurement process, aside from the variable being measured.

Question 8

What is a true score?

Answer 8

A score without measurement error, though it can’t be directly observed.

Question 9

How can an approximation of a true score be obtained?

Answer 9

By averaging multiple measurements.

Question 10

What are two influences that interfere with measuring a true score?

Answer 10

Time between measurements and the act of measurement itself.

Question 11

What is a carry-over effect?

Answer 11

When the measurement process alters what is being measured.

Question 12

How can practice effects in testing be minimized?

Answer 12

By using different sets of tests measuring the same construct.

Question 13

What is the fatigue effect in testing?

Answer 13

Reduced motivation or mental energy due to repeated testing.

Question 14

How is a true score related to a construct score?

Answer 14

A true score is tied to the measurement tool, while a construct score is tied to a theoretical variable.

Question 15

What is random error?

Answer 15

Unpredictable fluctuations and inconsistencies in the measurement process.

Question 16

What is systematic error?

Answer 16

Consistent or proportionate error related to the true value being measured.

Question 17

How is reliability related to variance in test scores?

Answer 17

Higher true score variance relative to total variance indicates higher reliability.

Question 18

What does a reliability of 0.80 imply?

Answer 18

80% of score variance is due to true differences.

Question 19

What is total observed variance in terms of reliability?

Answer 19

True variance + error variance.

Question 20

What does item sampling refer to in test construction?

Answer 20

Variations within items to measure a specific construct, such as physical aggression.

Question 21

What factors in test administration can cause measurement error?

Answer 21

Testing environment, test-taker conditions, and examiner behavior.

Question 22

What is test-retest reliability?

Answer 22

A reliability estimate based on the correlation between scores on two administrations of the same test.

Question 23

What is parallel form reliability?

Answer 23

Two identical versions of a test designed to measure the same construct with equal statistical properties.

Question 24

What is internal consistency reliability?

Answer 24

The consistency of items within a test, often measured with split-half or Cronbach’s Alpha.

Question 25

What does the Spearman-Brown formula do?

Answer 25

Adjusts reliability for split-half testing.

Question 26

What does a reliability coefficient of 0.7 signify?

Answer 26

An acceptable level of reliability according to Kline (2015).

Question 27

What is the purpose of a reliability coefficient?

Answer 27

To provide an index of error from test construction, administration, or scoring.

Question 28

What is the Classical Test Theory formula for observed score?

Answer 28

X = E + T (error + true score).

Question 29

What is Domain Sampling Theory?

Answer 29

A method estimating reliability based on sampling of test items and inter-item consistency.

Question 30

What does Generalizability Theory aim to describe?

Answer 30

Specific sources of variation in test scores due to testing conditions.

Question 31

What does Item Response Theory (IRT) analyze?

Answer 31

Item difficulty and item discrimination in assessing construct measures.

Question 32

What is Cronbach’s Coefficient Alpha used for?

Answer 32

It is used to measure internal consistency for tests with interval or ratio data, especially in Likert-scale formats.

Question 33

What range do values of Cronbach’s Alpha typically fall within?

Answer 33

Values range from 0 to 1, with higher values indicating greater reliability.

Question 34

What does a Cronbach’s Alpha of 0.99 suggest?

Answer 34

Potential redundancy in test items.

Question 35

What does a reliability coefficient of 0.80 mean in terms of measurement error?

Answer 35

20% of score variability is due to measurement error.

Question 36

What is the coefficient of inter-rater reliability used for?

Answer 36

To provide information about scoring error between multiple raters.

Question 37

What does the coefficient of stability assess?

Answer 37

Error due to time intervals between test administrations.

Question 38

What does the coefficient of equivalence assess?

Answer 38

Error from the instrument (test items) itself.

Question 39

What is a true score according to Classical Test Theory (CTT)?

Answer 39

The score that reflects an individual’s actual ability or trait level as measured by the test.

Question 40

What does Domain Sampling Theory suggest about test items?

Answer 40

Reliability increases as the number of items sampled increases.

Question 41

What is the goal of Generalizability Theory?

Answer 41

To identify specific sources of variation in test scores within the testing environment.

Question 42

What does “facets” refer to in Generalizability Theory?

Answer 42

Variables such as test item number, scorer training, and test purpose that define a test’s “universe.”

Question 43

What does Item Response Theory (IRT) focus on in terms of test items?

Answer 43

The difficulty level and discrimination ability of test items.

Question 44

What is meant by “level of difficulty” in IRT?

Answer 44

The degree to which an item is challenging or easy for test-takers.

Question 45

What does “level of discrimination” refer to in IRT?

Answer 45

An item’s ability to differentiate between individuals with high or low levels of the measured trait.

Question 46

How is reliability in a test related to true score and error variance?

Answer 46

Higher true score variance relative to total variance indicates higher reliability.

Question 47

What does a test with low reliability suggest about observed scores?

Answer 47

They are mostly determined by measurement error rather than true scores.

Question 48

How does the Classical Test Theory (CTT) view measurement error?

Answer 48

CTT assumes measurement error is always present in any test score.

Question 49

What happens to test-retest reliability over long intervals?

Answer 49

It tends to decrease, especially over intervals longer than six months.

Question 50

What is the coefficient of stability in test-retest reliability?

Answer 50

The reliability estimate for intervals over six months.

Question 51

How does parallel form reliability differ from alternate form reliability?

Answer 51

Parallel forms have identical statistical properties; alternate forms are similar but may vary slightly in mean and variance.

Question 52

What is inter-item consistency?

Answer 52

The degree of relatedness among items within a test.

Question 53

What does the Spearman-Brown formula help estimate?

Answer 53

Internal consistency reliability for split-half tests.

Question 54

When is the Kuder-Richardson Formula 20 used?

Answer 54

For determining inter-item consistency in dichotomous (binary) items.

Question 55

What is the main aim of Item Response Theory (IRT)?

Answer 55

To analyze item properties and assess the construct's ability being measured through item difficulty and discrimination.