3: Reliability

Question 1

Reliable measure

Answer 1

Produces consistent results when repeated measurements of the same quantity are made under identical conditions

• increases confidence of individual measurements
• well-defined and objective operational definition

Question 2

Test-retest

Answer 2

Consistent values every time you measure

Evaluating:
-r > or = .50 good

-composite score for extraversion
•remove items with poor internal reliability
• add responses to remaining internally reliable items on questionnaire, with reverse scored

-re-administer to same respondent at a later time
2 problems:
• respondent remember earlier answers(inflates r)
• respondents changing between administratations (deflates r)

When to use:
-appropriate for all but problematic for unstable variables

Question 3

Inter-rarter

Answer 3

Consistent values no matter who is observing

Evaluating:
Quantitative
-r > or = .7 good

Categorical

• % agreement = total agreements/ total observations
• 70-80%

When:
-observational measures that aren’t automated

Question 4

Internal

Answer 4

Consistent values no matter how you ask

-extent to respondents answer multiple questionnaire items designed to measure same construct consistently

Evaluating:
-cronbachs alpha
• from mean of all inter-item correlation
• alpha> or = .70 is good

When:
-self-report measures with multiple items

Question 5

Evaluating reliability

Answer 5

Empirical question

• paired measurements of same cases
• diff times, raters, or asked in diff ways

Quantitative:

• scatterplot
• correlation coefficient(Pearson r) (+)=high reliability

Categorical:
-Percent agreement

Question 6

Random measurement error (e)

Answer 6

Measured X= true value T+ bias B+ random error e

• measurements of same quantity Rarley be exactly the same due to random M. E.
• small random M. E. = more reliable

Reducing random M. E. :

• increases precision of measurement (reduces margin or error)
• increases stat power

Question 7

Systematic measurement error(bias) (B)

Answer 7

-average measured value systematically differs from the true value
(Apparatus; due to miscalibrarion, researcher, and subject)

Smaller= more accurate

-problematicfor frequency claims:
• can correct if know
• less prob. For association and causal, since stat power may be unaffected

Question 8

Accuracy

Answer 8

-produces results that agree with a known standard (closeness of measurement to true value)

-assess by measuring a known standard with the instrument
-less relevant when not using standard units
• can standardize a measure to have a specified mean and variability across a population

Question 9

Reliability

Answer 9

Closeness of repeated measurements to each other

• essential
• if unreliable a single measurement will vary unpredictably from tribe value even if accurate