Lecture 11_Measurement Error

Question 1

Measurement Error

Answer 1

failure of the recorded responses to reflect the true characteristics of the respondents

Question 2

Measurement error variance

Answer 2

Variance across respondents- social desirability- men diff answers than women-but ind person makes same errors
(y_i=μ+ε_i)
Variance across trials -diff answer each time for same survey
(y_it=μ+ε_it)

Question 3

Measurement error bias

Answer 3

When response deviation is non zero over conceptual replications

(y_i=μ+ε_i)

Question 4

Ways to Study Measurement error (5)

Answer 4

Repeated Measures
Lab investigations -non exp and exp
Interpenetration/random ass. To diff methods/modes
record checks; external sources
collecting correlates of error

Question 5

Repeated measures TYPES (4)

Answer 5

reinterview survey with subsample of survey sample
single int with repeated measures
combo of survey and admin rec’s
multiple rec’s collected by panel survey that refer to diff time pds

Question 6

Lab investigations - Non experimental

Answer 6

Expanded interviews (Think aloud, Comprehension probes, Debriefing)
Methods to target source of error (vignettes, response timing)
Confidence measuresBehavior coding

Question 7

Issues with lab investigations

Answer 7

Attention level of Respondents
Motivation in response process
Type of participants
Reality

Question 8

random ass. To diff methods/modes

Answer 8

Split ballot; split sample within surveys
Difference between T groups used to est biasMost focus on bias not Measurement error variance
Inference is to the sampling frame

Question 9

Assumptions of lab investigations

Answer 9

Reality

Heterogeneity of errors represented

Question 10

Interpenetration definition

Answer 10

Randomly assign subsamples to different production units (interviewers)
Many diff treatments are applied and compared- more common to focus on variance
All survey attributes are replicated for diff int. Estimation of variance due to production units (interviewers)
Interviewer variance

Question 11

Interpenetration model

Answer 11

y_ij=χ_i+M_ij+ε_ij

χ_i:true
M_ij:effect of jth method

Question 12

Issues with interpenetration/Interviewer variance models

Answer 12

Changes survey conditions
Logistical issues -(Attrition, Refusal conversion ass. Of interviewers)
interpreted to mean that reduction in assigned cases (m) leads to lower int effects

Question 13

Record checks; external sources Assumptions

Answer 13

Most interested in bias NOT measurement error variance

Record is assumed true value

Question 14

Record checks; external sources TYPES

Answer 14

Reverse record check
Forward record check
*Full design record check

Question 15

Reverse record check

Answer 15

Sample people with known characteristic covered in record

Measures failure to report on something E.g., abortions, crime incidents

Question 16

Record check model

Answer 16

y_i=R_i+ε_ι

R: record value for reporter

Question 17

Issues with Reverse record check

Answer 17

Cant measure overreporting (only a subset of events sampled)

Question 18

Forward record check

Answer 18

Samples reporters from a frame; then compares survey answers to recorded value
Measures overreports (when survey report not found in record)E.g., medical events

Question 19

Issues with forward record check

Answer 19

If no report is given; no record search conducted

Question 20

*Full design record check

Answer 20

Requires full population covered by one or more record systems; full access
All events captured
Asking about variables related to the events that are available for population of interest
Measure both under and over reporting E.g., official divorce records

Question 21

Issues with record checks

Answer 21

Assumes all records are complete

Linking errors

Question 22

Using Benchmarks

Answer 22

Universe data available to compute the same statistic as the sample
e.g., voting

Question 23

correlated response variance

Answer 23

interviewer variance

answers from one respondent are correlated with answers interviewed by the same interviewer

Question 24

simple response variance; A.K.A. reliability in psychometrics

Answer 24

respondent vary in their response deviations over conceptual trials anyway- even without interviewer effects

Question 25

Index of consistency

Answer 25

proportion of the element variance of y that’s due to the variance of the response deviations

Var(ε)/Var(y)

Question 26

Measure response variance

Answer 26

Y_i1=χ_i+ε_i1
Y_i2=χ_i+ε_i2

Variance in errors

E=(ε_i1-ε_i2)

Question 27

psychometric “reliability”

Answer 27

p_y=1-index of consistency

Question 28

Assumptions to measure Simple Response Variance

Answer 28

No change in true values, over X trials (threatened by long time between trials)
No memory effect of first trial on second (threatened by short time between trials)
Survey conditions the same between trials
(threatened by using supervisors to conduct int’s)

Question 29

Parallel measures

Answer 29

survey items that measure the same underlying construct with different response deviations
Both measures are asked of each respondent as indicators of same underlying attribute
Correlation should be high

Question 30

Generalizing parallel measures/multiple indicators

Answer 30

if multiple constructs measured and different methods are used for each- then separate out variance into
True values
Method effects
random measurement error variance

Question 31

Challenges of using multiple indicators

Answer 31

Context effects- similar questions
estimation issues with discrete variables
“just-identified” models

Question 32

Paradata in surveys

Answer 32

data collected as a byproduct of field activities

• response latency
• keystroke data
• probng behavior
• edit failure records