SOC200 - Surveys (Chapter 8) Flashcards
SURVEY RESEARCH
form of research in which data are collected through predetermined set of questions administered to a sample
Ideally, the sample comes from your population of interest
Observation and analysis occur together
Main Purpose of Survey Research
To generalize findings from sample to pop of interest
Medium for data collection
Probability sampling – generalizable results
You are never gonna get a perfect sampling frame that includes all of target pop
Just make it as inclusive as possible
Will always have errors
When is Survey Research Appropriate?
for all 3 main types of research (descriptive, exploratory, explanatory)
Good for when researchers want to know something about pop that is too big to observe directly
When is Survey Research Appropriate?
Excellent for measuring attitudes + orientations in large pop (political polling)
When done right, some of most valuable tools for social inquiry
When is Survey Research Appropriate?
Just limited by types of questions you can include
Effective of getting accurate view of characteristics
Economic + efficient
Western World is Suffering from Survey Fatigue!
Constantly bombarded with surveys + data collection
No time + energy
Drive in collecting through electronic habits without consent
Qualities of a “Good” Survey: The appropriate use of questions and statements
“I drink hard liquor…”: Less than once a week, 1 to less than 5 times a week, 5 to less than 10 times a week, 10
times or more a week
Qualities of a “Good” Survey: The appropriate use of questions and statements
don’t work well: drink hard liquor: awkward. Better as question
sometimes better as a question, sometimes as a statement
Obama question: too long
Qualities of a “Good” Survey: The appropriate use of open-ended and closed- ended questions
How many times a week do you drink hard liquor? – ambiguous, ppl can interpret it in many ways
How many times a week do you drink hard liquor? - 1-5 times a week, 5-10 times a week, 10 times or more a week
Qualities of a “Good” Survey: The appropriate use of open-ended and closed- ended questions
how many times a week do you drink hard liquor? – no N/A, answers not mutually exclusive
BETTER: In the last 7 days, how many drinks containing distilled alcohol have you consumed?
Qualities of a “Good” Survey: Clearly and unambiguously worded items
How many car repairs did you do last year?
Do they mean car repairs that I personally did, or that my mechanic did? Do they mean “last year” as in the calendar year, or do they mean in the last 365 days from today?
Qualities of a “Good” Survey: Clearly and unambiguously worded items
What is your current marital status?
I think it’s great. But my wife, well… Who knows what she’s thinking! What do they mean by marital status? Do they mean whether my marriage is happy, sad, or whether I’m currently single, divorced, or widowed?
Qualities of a “Good” Survey: No “double-barrelled questions”
How would you rate the quality of the instruction in SOC 200?
How happy are you with your phone provider’s rates and customer service?
PROBLEM: one question is asking you about two different things (rates and customer service) – can be misleading
can have diff views on both parts
if it contains “and” – asking 2 things
Qualities of a “Good” Survey: No “double-barrelled questions”
Do you think universities should lower tuition rates in order to increase post-secondary enrolment?
PROBLEM: question still asking about two things (should universities lower tuition rates + asking if you think lowering rates will increase enrolment)
Qualities of a “Good” Survey: Asking questions that respondents are competent to answer
Approximately how many bottles of beer have you consumed in your life?
Do you think that China’s Three-Gorges Dam project will be beneficial for its environment?: who knows this?
Qualities of a “Good” Survey: Asking questions that respondents are competent to answer
How many times do you blink your eyes in a day?: no one can count that high
What was the first brand of toothpaste you started using?
Optimal recall periods: when is the best time to get these answers
Is it relevant?
Qualities of a “Good” Survey: The questions are relevant to the population of interest
What is your favourite meal using pork?: ppl with dietary restrictions – not appropriate
Do you think that China’s Three-Gorges Dam project will be beneficial for its economy?
Qualities of a “Good” Survey: The questions are relevant to the population of interest
On a scale of 1 to 10, with 10 being the most challenging, how would you rate your ability to balance your work and family responsibilities?: depends on the age range, probably not for us
Which of the following newspapers do you read most frequently?: replaced by internet
Qualities of a “Good” Survey: Having respondents that are willing to answer
Some questions may be too sensitive/embarrassing for particular demographic to answer/answer honestly
One recent study found that while U.S. respondents skip sensitive questions, Asian respondents often discontinue survey entirely
Culture + race has also been linked to response styles:
Differences in extreme responding have been found between a variety of national + ethnic groups
Qualities of a “Good” Survey: Having respondents that are willing to answer
tendency of some respondents to agree with questions regardless of content, norm of agreeableness, varies significantly by culture
Dealt with through question wording; ensuring anonymity; providing item response choice that allows respondents to say “prefer not to answer”
Qualities of a “Good” Survey: Shorter items are better
The respondent should be able to read the item quickly, understand its intent, and provide an answer without too much study
Reduces survey fatigue
Minimizes the number of potentially misleading answers because the question was too complex for the respondent to understand
Qualities of a “Good” Survey: Shorter items are better
May reduce level of non-response/number of uncompleted survey items
keep the items simple, questions clear
harder to do in practice rather than in theory
too short: not communicating enough of question, background info
Qualities of a “Good” Survey: Questionnaires that avoid negatively worded items
Negation = easy misinterpretation
instructor did not respond to students’ comments during class: Strongly Disagree, Disagree….
might skip/be confused
Qualities of a “Good” Survey: Questionnaires that avoid negatively worded items
-
Qualities of a “Good” Survey: No leading questions, jargon, vague concepts, or biased terminology
More people own an iPhone than any other smartphone. Do you own an iPhone?
leading: marketing strategy – get on bandwagon
setting you up to answer untruly
may feel embarrased
social desirability bias
Qualities of a “Good” Survey: No leading questions, jargon, vague concepts, or biased terminology
Indicate your level of agreement with the following statement: “My recent vacation was just peachy”: no colloqualisms
How many times have you ever wanted to hit your wife?: scare respondent into saying 0 times, assumes that only men hit spouse, assumes not relevant to unmarried ppl – makes too many assumptions
Designing Effective Questionnaires
Questionnaire format is extremely important for ensuring high response rates and accurate answers
Questions should be spread out and uncluttered
No more than one question on a line
Cluttered questionnaires can:
Confuse the respondent and the interviewer
Designing Effective Questionnaires
simple questionnaire format will mean positive experience for respondent
easy to follow
hard to code + input data set
internet survey: control flow of research
Example of a Good and Bad Questionnaire Format
trial: matrix format - complicated, not clear to read, jumbled, arrows confusing
revised: improved navigation, nice shading, get away from matrix, easier to comprehend, nicer to look at
Item Response Formats
Boxes/circles for checking, placed underneath the question + ordered vertically
3. Code #: for circling, placed underneath the question, and
way we understand + process survey need to account for: right to left for chinese/arabic
Item Response Formats
ordered vertically, specifying code # to be inputed
interviews know # is circled, not crossed out
data entry person would know what # to do
can use scantron
Types of Question Formats: 1. The Contingency Question
whether question is asked + answered depends on response to first question in the series
set off to side + enclosed in box + arrow leads to next question
helps to put instructions at top of page to remind ppl to answer questions that apply to them
computer based: next question appears automatically
Types of Question Formats: 2. The Matrix Question
ask several questions that have same set of answer categories
Uses questionnaire space efficiently
Faster for respondents to complete
Allows respondents to easily compare answers with earlier questions
response set: when you see this, they rip right through it without much though
statements must be short + clear
Importance of Question Order
order of questions can have a substantial impact on:
overall survey response rate
# + type of questions answered
accuracy of provided responses
sensitive questions at beginning discouraging
demographic data at beginning – routine form – no motivation
Importance of Question Order
more interesting questions at beginning so they glance + want to do it
respondents aim to be consistent with preceeding responses
randomized order not practical - unorganized
just be sensitive to effects of ordering
gaining rapport – start with non threatening common questions then move on to sensitive stuff later on
self administered: sensitive stuff later on
Pretest
Imagine spending hundreds of thousands or millions of dollars on survey design, only to discover that poor response rates and misleading answers were caused by:
ambiguously worded questions and item responses
format of questionnaire + ordering of questions
survey that was not relevant for the intended sample
Pretest
See how ppl respond
Did they give answers you weren’t expecting
Important they understand what you’re doing
Work out the bugs
Methods of Collecting Survey Data: 1. Self-administered Questionnaires
Hard copies delivered in person/through mail/soft copies administered online via links to a website
monitor returns + ensure anonymity
graph – detect response bias, when, where, answers
properly timed follow ups – 2-3 week gaps
online questionaires similar: access those with internet – may not be generalizable
Methods of Collecting Survey Data: Live Interviews
via face-to-face/live interviewing by telephone
demeanour: composure, language
how they dress + act must not influence responses
high response rate – harder to say no to someone
able to help fill out questionnaire – provide clarification
probe: minimize idk + prefer not to answer
not following script, dress (inappropriate, unprofessional) – bias/wilingness to participate
feel more guarded when talking to person
limit to what they can ask – may be untruthful
Methods of Collecting Survey Data: Telephone
Via computer assisted telephone interviewing (CATI)
telephones used to be a luxury – built in bias
reaching respondents from certain class – answers reflect bias
access widened – representativeness increased
dropping landlines for cellphones, caller ID
nonresponse bias – less representativeness
Example of a Survey Response Rate Graph
week 1: high response rate
drops off
follow up: spikes up again
those inclined to response = those who would reelect rob ford
track reaction to diff news stories – spike in disapproving responses who saw the news
track biases + reactions to events
Interview Administered Questionnaire
Demeanour, familiarity with questionnaire + following question wording exactly key to successful interview administered questionnaire
Telephone Surveys
Computer Assisted Telephone Surveys (CATI)
Self-administered Questionnaires
Cheaper & quicker than face-to-face interviews
Require small staff
Higher response rate to sensitive questions
Makes large samples feasible
Self-administered Questionnaires
Lower response rates
May have to limit survey to questions minimally appropriate for all respondents
Interview-based surveys
More effective for complicated surveys
Can conduct survey based on sample of addresses/phone numbers
Face-to-face interviews allow relevant additional observations
Better at dealing with context of social life
Interview-based surveys
More expensive, especially for national survey
Training + quality assurance of interviewers adds to reliability issues
Telephone surveys
Saves time + money, relatively far-reaching for cost
Less risk to interviewer’s safety
Impact of interviewer on responses is lessened
Telephone surveys
Easy for respondents to ignore, increasing non-response bias
interviews: lower response rates to sensitive questions
Online surveys
Same strengths as self-administered questionnaires, but cheaper
Online surveys
No way to ensure representativeness of some general pop
May be impersonal + thus easily ignored, leading to lower response rates
