Week 8 : Survey Research Flashcards
1
Q
What is a survey?
A
- highly structured… close ended questions & fixed response options
- quantitative
- large sample size
- large number of questions
2
Q
Why survey? (3)
A
- quantify change over time
- effectiveness
- breadth of topics
3
Q
Why survey?
1 - quantify change over time
A
- within-person change
- cohorts
- societal trends
- done through… cross sectional & longitudinal (repeated cross sectional & panel designs)
- MUST ask same questions every time/place when doing longitudinal
4
Q
Why survey?
2 - effectiveness
A
- standardized formats (consistent measures)
- short questions & multiple choice answers
- large sample possible
- representativeness (generalizability)… high external validity if used a probability sample
5
Q
Why survey?
3 - breadth of topics
A
- can include many questions/topics
- easy to replicate
- data for many studies (can be used by researchers with diverse research interests)
6
Q
Primary vs secondary data
A
- primary data… social scientists can design & carry out their own surveys
- secondary data… use & analyze data from large surveys carried out by others like the government & large universities
7
Q
Challenges of survey research (4)
A
- nonresponse
- coverage error
- sampling error
- measurement error
8
Q
Challenges of survey research (4)
1 - nonresponse
A
- Survey nonresponse… no participation at all
- Question nonresponse… no response to particular questions on the survey
- Systematic error… individuals who do and do not respond differ in systematic ways! (bias)
9
Q
challenges of survey research (4)
1 - nonresponse solutions
A
- anticipate potential errors with a literature review
- write good survey questions
- organize the questions (professionalism is key!)
- test & revise the survey questions before launch
10
Q
challenges of survey research (4)
2 - coverage error
A
- the sampling frame does not adequately capture all members of the target population
- either by systematically omitting some or by including others more than once
*
11
Q
challenges of survey research (4)
2 - coverage error solutions
A
- Anticipate the potential errors (lit review)
- Address the errors in sampling strategy
- Acknowledge any systematic errors
12
Q
Challenges of survey research (4)
3 - sampling error
A
- sampling does not represent the population accurately
*
13
Q
Challenges of survey research (4)
4 - measurement error
A
- occurs when the approach used to measure a particular variable affects the response provided
- Source of error is the survey design, the interviewer, the setting, etc.
14
Q
4 modes of administration for surveys
A
- face-to-face
- telephone surveys
- mail or Self-administered questionnaires (SAQs) surveys
- online surveys
15
Q
Criterea for deciding which mode of adiminstration to use
A
- cost
- response rate
- researcher control over data-collection process
- potential for interviewer effects
16
Q
1 - face-to-face
A
- interview schedule
- high cost, response rate, researcher control & potential for interviewer effects
- interviewers make observations about the setting too (para-data)
- ensures participants answer more honestly about like height & weight and visible things like that
17
Q
2 - telephone surveys
A
- Structured interview administered over the phone
- moderate cost, high response rate, moderate researcher control & potential for interviewer effects
- pros… interviewer can guide, less planning, can guage opinions on emerging issues
- cons… interviewer effects, fatigue, cell phones, ppl don’t wanna be bothered & no showcards (to remember responses)
18
Q
3 - Mail or Self-administered questionnaires (SAQs) surveys
A
- low cost, response rate, researcher control & potential for interviewer effects
- questionairre
- cons… more missing data/idks, can’t ensure targetted repsondent acc filled it out & very low response rate
19
Q
4 - Online surveys
A
- low cost, moderate response rate, moderate researcher control & low interviewer effects
- pros… little risk of data entry errors, easy nevigation & reach a lot of ppl
- cons… ‘diginal divide’ (access), can’t be probability sample, ethical concerns maybe
20
Q
Sampling for surveys…
A
- the goal is to quantify while elimintation systematic errors & using stats to estimate sampling errors
- probability samples… stratified sampling good but very expensive
- non-probability samples… systematic bias (Wier et al.), useful but not really
21
Q
Sampling for surveys - quota sampling
A
- Convenient sample (non-probability)
- Can have quota to mirror population characteristics
- Quota ≠ Strata
- e.g. online sample services , qualtrics, Amazon M Turk, etc.
- more affordable that probability samples
- Still: systematic error!
22
Q
ethical considerations…
A
- Most surveys are not sensitive, but some are! (e.g. HIV, drug use, sexual orientation)
- confidentiality… surveys obtain highly personal and sensitive information & if this information is made public, it could be a source of harm to respondents (use identification IDs)
- anonymity… unrealistic & unpractical, can’t do follow-ups if they can’t figure out who they are
- can cause minor psychological distress
23
Q
Typical steps to survey research
A
- research questions
- conceptualization & operationalization (Theory -> Hypotheses -> Concepts….. -> Variables)
- sample & survey mode
- survey questionairre (write & organize)
- data collection
- data processing
- analysis
24
Q
Survey content (organization)
structure of questions
A
- The stem poses the question & the response categories provide the pre-set answers that the respondent may select
25
Q
Survey content (organization)
Closed-ended vs Open-ended questions
A
- closed ended… Answer options must be mutually exclusive & exhaustive, often used to measure the frequency of behaviours & many use a rating scale
- open ended… only require a brief answer, more personal, can be acc question or just ‘other: (specify)
26
Q
survey content
organization of questions…
A
- skip patterns & screener questions
- establish trust & rapport early in the survey
- avoid monotony & response set (switch between positively & negatively worded questions)
- avoid order effects (improve by adding sections)
- avoid priming effects
27
Q
survey content
Single Item vs Composite…
A
- single item… one variable = one survey question
- composite variable… one variable = many survey questions & also need to specify how you will combine them (either indexing or scaling)
28
Q
Characteristics of High-Quality Questions
4 characteristics of high-quality questions…
A
- use clear & simple language
- be precise
- exercise caution when dealing with sensitive topics
- avoid leading questions
29
Q
Characteristics of High-Quality Questions
1 - use clear & simple language
A
- brief, clear & concise
- no academic jargon
- double negatives should be avoided cuz they are difficult to understand (e.g. “do you favor or oppose not legalizing physician- assisted suicide?”)
30
Q
Characteristis of high-quality questions
2 - be precise
A
- avoid double barelled questions…
- e.g. “How much confidence do you have in the Prime Minister’s ability to handle domestic and foreign policy”
- do not assume prior knowledge of respondent
31
Q
Characteristis of high-quality questions
3 - caution with sensitive topics
A
- use neutral language that does not stigmatize the behaviour of interest
- questions should avoid emotionally laden words that might upset respondents
32
Q
Characteristis of high-quality questions
4 - avoid leading questions
A
- questions should be as fair & as even-handed as possible
- veading questions are phrased to lead or guide responses to a particular answer, yielding results that support the organization’s goals
- may include a morally charged word that pushes respondents toward or away from a particular answer
- avoid acquiescence bias, in which some respondents answer ‘agree’ to attitudinal questions (yes/no options)
33
Q
recap
A
- Thoughtful sampling can help a lot with coverage error and sampling error
- Post-survey weighting can help with survey non- response
- Some statistical procedures can help with question non- response (missing values)
- But a large part of survey success lies in the questionnaire
○ Pretest
○ Preliminary data analysis
Cognitive interviews
