L5 Survey Research Flashcards
What is survey research?
Research, based on a questionnaire to which respondents record their answers typically with closely defined alternatives
What is the ICS?
Index of Consumer Sentiment:
focused on past and current personal and general economic conditions
What is CBS?
Centraal Bureau Statistiek:
Focused, among others, on demographic, social and economic statistics
When to use surveys?
- When you want to say something about a population, but you cannot measure the whole population (very often)
- When you are interested in quantitative descriptors
- personal measures (archival is mostly only objective and not subjective)
What are the survey research design decisions?
- Operationalization of concepts (variables)
- Decide on survey mode
- Appearance of the questionnaire
- Data collection
What is operationalization
- Reduction of abstract concepts
* to render them measurable in a tangible way
What are off-the-shelf measures?
usen before and proven succesful by prior researchers
What are open-ended questions?
Allows respondents to answer a question in any way they choose
(+)No limit to possible responses, sometimes participants prefer own words
(-) More difficult to respond and analyze
What are closed-ended questions?
Asks the respondents to make choices
among a set of alternatives
(+) Help respondents to make quick decisions
(-) Alternatives need to be mutually exclusive and collective exhaustive
Single-item measures
When concrete singular object/attribute:
- maritial status
- profession
Multi-item measures
When measuring more abstract questions
In all other cases
• Use ‘off-the-shelf’ scales
• Develop your own scale
Mixed-mode designs
a combination of different survey models, leads to a trade-off costs and errors (e.g. web survey + mail to seniors)
Advantages and disadvantages of Off-the-shelf scales
(+) • Known and “good” validity and reliability • Comparability of results • Low cost
(-)
• Not tailored to your exact
research need
• Require translation if in different language (source of error)
Double barreled questions
question that lends itself to different possible responses to its subparts (2 subjects in one)
“How is the taste and appearance of your pancakes?”
ambiguous questions
the respondent may not be sure what the question exactly means
“Are you happy?”
Leading questions
lead the respondent to give the responses that the researcher would like them to give
“A true American’s favorite colors are red, white and blue. Are these your favorite colors?”
Loaded question
question phrased in an emotionally charged manner
“have you stopped cheating on tests?”
“do you still beat your wife”
Double negatives
unclear what people mean when answering yes/no
“Do you oppose not allowing the board to pass article 10 of the ballot?”
what are comparative scales (ranking scales)?
- Paired comparison
- Rank ordering
- Constant sum
Paired comparison
Now you will see ten pairs of shampoo brands. For each pair, please indicate which one of the two brands you would prefer for personal use
Rank ordering
Rank these brands in order of preference for personal use.
- Cola ___
- Ice Tea ___
- Fanta ___
Constant sum
Divide 100 points among the following brands, indicating your preference for personal use:
- Cola (points)
- Ice Tea (points)
- Fanta (points)
What are non-comparative scales (rating scales)
- Continuous rating scale
- Likert scale
- Semantic differentials
Continuous rating scale
- “How would you rate De Bijenkorf as a department store?”
* 0 ———————————- 100
Likert scale
- Disagree/Agree (with statements)
- 5- or 7-point
Semantic differentials
“What do you think of the Apple iPhone?”
• Good / bad
• Powerful / weak
• Modern / old-fashioned
mutually exclusive
Only one answer applies
collectively exhaustive
the answer possibilities cover the entire realm of possible answers
Survey modes
How the data are collected
Factors that play a role when deciding on survey mode
- Measurement
- Representation
- Economics
Appearance of the questionnaire
- A good introduction
- Organizing the questions & giving instructions and guidance
- Personal data (do not scare them off)
- Concluding the questionnaire
- also include confidentially
Goals of pretesting
- Catch errors/unclarities
- Discover sensitive topics
- Check response categories
- Optimize length
How?
• Pick 5-10 people from target group
• Let them “think out loud” + observe
• Improve survey
Response rate
of people that participated in the survey divided by # of people sampled
is generally low
• Internal surveys: 30-40%
• External surveys: 10-15%
How to increase your response rate?
Maximize rewards of participation
• Showappreciation
• Use interesting/friendly questionnaires
• Offer tangible rewards
Minimize costs of participation
• Minimize time and effort required
• Minimize chance of feeling threatened by questions
Maximize trust
• Ensureanonymity/confidentiality
• Open lines of communication with participant
• Identify research with well-known, legitimate organization
also provide contact info
What is validity?
- Does an instrument measure what it was intended to measure?
- Do we measure what we are supposed to measure?
What is reliability?
Are the data accurate (free from measurement error) and consistent (from one occasion to another)?
Measurement validity
- Provide precedence (other studies same measure)
- Provide sound logic to support that considerable conceptual overlap exists between measurement/proxy and construct
- Be aware: single-item measures for abstract constructs = low validity
Proxy
proxy= indirect measure of desired structure.
commonly used when direct measures are unavailable
social desirability bias
Respondents may not always be willing to communicate their true response in case of sensitive issues
- drugs use
- finances
How to minimize social desirability bias
- “Everybody-does-it”
- “Assume-the-behavior”
- “Authorities-recommend-it”
- “Reasons-for-doing-it”
in all other cases, avoid leading and loaded questions
Survey mode effects
Participants are more likely to give socially desirable responses on the phone or face-to-face
How to measure multi-item?
Cronbach’s alpha
What does Cronbach’s alpha do
- Cronbach’s alpha measures to what extent a set of items are inter-related
- High inter-relatedness = high reliability
- Cronbach’s alpha is between 0 and 1
- Values > .7 are considered acceptable
