Questionnaire Design Flashcards
Main Decisions when Designing Research Instruments
- Type of Questions
- Wording of the questions; how to ensure that they are clear; what are the possible answers
- Design of the questionnaire (set the right order of questions and answers); check for the instructions
- Wording of an accompanying letter
- Method of distribution and return of the completed questionnaires (how the data collection mode limits the questions posed)
- Methods for collecting the data; actions to be taken if questionnaires are not returned
Question Content (3 things)
- Behavior
* Want to know what the respondents are doing
* When you interpret the findings, NEVER assume it’s a direct reflection of their behavior
* Verbal report - they only think or say that they go twice per week
* Theoretically we would have a chance to double check information
* Response bias; social desirability - Beliefs, Attitudes
* How satisfied are you with
* Cannot double-check; have to rely on responses - Characteristics, Attributes
* Gender, age, marital status, highest education, etc.
* These questions aren’t part of your findings
* Without demographic info, you can’t see the difference in attitudes between groups
* It’s really easy to forget these demographic questions - don’t forget
Types of Questions (4 types)
Direct vs. Indirect
Open-ended vs. Closed
Direct Questions
- Ask the respondent directly what’s their opinion; using “you”
- What brand do you like?
- Most questions should have a direct form
Indirect Questions
- Projective questions - qualitative
- Which brand do people like?
- When asking about sensitive information, indirect questions might be better
- Must be careful when interpreting because answers are not straightforward
Open-ended Questions
“Questions for which the respondent is asked to provide his or her own answers. In‐depth, qualitative interviewing relies almost exclusively on open‐ended questions. See Chapters 8 and 11.”
- Allow respondents to present their own answers; no options
- Give answers in their own words; use their own knowledge
- More difficult to respond to these questions
- Enables us to see how they formulated those answers
Closed Questions
“Survey questions in which the respondent is asked to select an answer from among a list provided by the re‐ searcher. Popular in survey research because they provide a greater uniformity of responses and are more easily processed than open-ended questions. See Chapter 8.”
- Provide wording of question and answers for respondents to choose one
- Yes/no questions; multiple choice
Open-ended Question Disadvantages
- Many respondents skip these questions
- If it’s part of an interview, the first response is “I don’t know”; researchers must probe of
- More difficult for researchers to analyze; categorize answers; code them
Types of Open-ended Questions (2 categories & 5 types)
Infinite responses
vs.
Limited amount of answers
- Starting questions
- Follow-ups
- Grand Tour
- Questions bringing arguments
- Questions bringing attitudes
Infinite responses
What do you think of our current president?
Limited amount of answers
What brand of yogurt do you get?
Starting questions
What role does TV have in your everyday life?
- Gets people thinking about the topic; focus the respondent’s attention to the topic
Follow-up questions
What exactly do you remember from the advertisement?
- Can distinguish liars; if they did see the ad, they could give something specific; if they didn’t see the ad, they’d give something general or incorrect
- Without a follow-up, you’d get biased info
Grand Tour questions
Please describe your usual activities during a workday?
- Could give a lot or a little
- Want to know how the respondent presents activities; how much info they can provide for the broad question
- Want to see the emphasis they put on each part
Questions bringing arguments
Why did you purchase the product?
- Makes person defend themself; Gives benefits of the product
Questions bringing attitudes
What do you think about the President?
- Not about the object but the subject; asking about that person’s opinion
Types of Closed Questions (6 types)
- Lists
- Categories
- Rankings
- Scales
- Quantities
- Grids
List Questions
Which of the following research activities have you engaged in the last seven days?
If you haven’t engaged in a particular activity, please leave this box blank:
(check all that apply)
[ ] Accessed an online database
[ ] Accessed the internet
[ ] Read a journal article
[ ] Conducted an interview
- Interviewers should read the question, then read each question, letting the respondent answer yes/no individually before moving onto the next item in the list
Category Questions
In the last month, how often have you visited the library?
(choose only one option)
[ ] At least once a day [ ] At least once a week [ ] At least once every two weeks [ ] Once within the last month [ ] I have not visited the library in the last month
- Easy to respond
- Here, read all the response items and ask them to respond after
- 3-5 options; more not recommended
- Should create a “showcard” with all options that you can give/show to the respondent
Ranking Questions
Please rank the following in order of importance to you in relation to your MSc studies. The most important should be given a rank of 1, the next most important rank of 2, etc. If you feel that a factor is of no importance, please leave it blank.
[ ] the course material [ ] the library facilities [ ] the computer facilities [ ] the lecturing staff [ ] assessment methods
- People can usually identify most important but have trouble with later on the list
Scale Questions
XY has been an excellent Prime minister 1 - Strongly agree 2 - Agree 3 - Neither agree nor disagree 4 - Disagree 5 Strongly disagree
- Type of category question
Quantity Questions
How many times have you been to the USA? ___
What year were you born? ___
- Open-ended
- Response is always a number
Grid Questions
For your research work, how often do you use the following type of information source?
- List of categories in the left column followed by a scale next to each item, for example 1 (Never) to 7 (Very frequently)
- Saves space and time on questionnaires
- Avoids repetition of (almost) the same question for multiple items
Common Mistakes in Questionnaires (6 types)
- Double barrel
- Generalizing
- Mismatched wording
- Branching
- Mutual exclusivity
- Variability
- Uncertain options
Double barrel mistake
Question:
How many cups of coffee or tea do you drink in a day?
- Not objective - how big is your cup?
- May not drink coffee or tea
- May drink more/less on different days
- There are coffee drinkers and tea drinkers
Principle:
- Ask for an answer in only one dimension
- Double barrel question
Solution:
* Should be separated into two different questions:
- How many cups of coffee do you drink in a day?
- How many cups of tea do you drink in a day?
Generalizing mistake
Question:
What brand of computer do you own?
a. IBM PC
b. Apple
- Missing lots of brands
- May not own a computer
Principle:
- Avoid hidden assumptions
- Make sure to accommodate all possible answers
Solution:
- Make each response a separate dichotomous item
a. Do you own an IBM PC? (yes/no)
b. Do you own an Apple computer? (yes/no) - Add necessary response categories and allow for multiple responses
a. What brand of computer do you own? (circle all that apply)
* IBM PC
* Apple
* Other
Mismatched wording mistake
Question: Have you had pain in the last week? [ ] Never [ ] Seldom [ ] Often [ ] Very often
Principle:
Make sure questions and answer options match
Solution:
Reword either question or answer to match
How often have you had pain in the last week? [ ] Never [ ] Seldom [ ] Often [ ] Very often
Branching mistake
Question:
- Do you currently have a life insurance policy? (Yes or No; If no, go to question 3)
- How much is your annual life insurance premium?
Principle:
Avoid branching to avoid confusing respondents
Solution:
Write as one question
1. How much did you spend last year for life insurance? (Write 0 if none)
- If people see repetitive branching in questions, people will adjust their answers to avoid the pain in the ass of answering the confusing questions
- Speeds up interview
Mutual exclusivity mistake
Question: Where did you grow up? a. Country b. Farm c. City
Principle:
Avoid questions having non-mutually exclusive answers
Solution:
Design the question with mutually exclusive options
Where did you grow up?
a. House in the country
b. House in the city
c. Farm in the country
Variability mistake
Question:
Are you against drug abuse?
Principle:
Write questions that will produce variability in the responses
Solution:
Eliminate the question
- Social desirability effect - nobody supports drug abuse
- Usually used to prove a point in data collection
Uncertain options mistake
Question: Which one of the following do you think increases a person’s chance of having a heart attack the most? (Check one) [ ] Smoking [ ] Being overweight [ ] Stress
Principle:
Encourage the respondent to consider each possible response to avoid the uncertainty of whether a missing item may represent either an answer that does not apply or an overlooked item.
Solution:
Which of the following increase the chance of having a heart attack?
Smoking:
[ ] Yes
[ ] No
[ ] Don’t know
Being overweight:
[ ] Yes
[ ] No
[ ] Don’t know
Stress:
[ ] Yes
[ ] No
[ ] Don’t know
- Don’t know is an important piece of information that you wouldn’t get from “check all that apply” instruction from the first question; distinguishing “yes/no” from “don’t know”
Mistakes overview
- One question may have many types of mistakes
- Some questions were taken from real questionnaires
- Some things are wrong every time; some items are mistaken only based on the context of the rest of the questionnaire
- You can reuse questions from a questionnaire, but be careful - it could make sense and be a good question based on the context there, but it might be a mistake in the context of your own
- How many cups of coffee or tea do you drink in a day?
- Originally taken from questionnaire trying to assess consumption of cold drinks vs. hot drinks
Pilot Study Benefits
- Allows problems in questionnaire to be identified & corrected
- Predict how respondents will interpret & react to questions
- Opportunity to refine & develop interviewing skills
- Highlight possible sources of interviewer bias
- Provides feedback on respondents’ interest in the topic
- Identifies the mistakes in instructions
Impact of Numerical Labels
- People attend to incidental features of questions, such as numerical labels
- Responding on a scale of -5 to 5 vs a scale of 0 to 10
- Uncommon scale can skew answers since people have to think much harder
- May assert negative connotation on lower scores
Sources of Response Bias
- How the order in which the alternatives are listed affects the distribution of replies
- Slightly higher responses for items that are ordered first
Much greater (5%) Somewhat greater (9%) About the same (48%) Somewhat less (26%) Much less (12%)
VS.
Much less (19%) Somewhat less (23%) About the same (46%) Somewhat greater (10%) Much greater (2%)
Face vs. Radio Button
- NPS Scores
- Overall, how do you rate [journal name]?
Fixed (smileys): 1 - very good - 12.1% 2 - good - 36.4% 3 - satisfactory - 36.4% 4 - adequate - 10.1% 5 - inadequate - 4% 6 - unsatisfactory - 1%
Radio buttons: 1 - very good - 4.6% 2 - good - 52.8% 3 - satisfactory - 31.3% 4 - adequate - 7% 5 - inadequate - 3.2% 6 - unsatisfactory - 1.1%
- bigger distribution with smileys perhaps because of the images
Context and Images
- Images added to illustrate questions may have unintended effects on interpretation of question
- Images are necessarily concrete and may affect construal of category
- Images are powerful contextual stimuli
Example:
How often have you been shopping in the last month?
(Picture of grocery store)
vs.
(Picture of clothing store)
Before you start a questionnaire
Be very clear about what you need to learn
- What are the questions you have about your subject of study?
- What questions emerge from your objectives?
Checklist for Eliminating Questions
- Does the question measure some aspect of the research problem?
- Does the question provide info needed in conjunction with some other variable?
- Will most respondents understand the question? In the same way?
- Will most respondents have the information to answer the question?
- Is the instruction correct?
- Will most respondents be willing to answer it?
- Is other information needed to analyze this question?
- Should this question be asked of all respondents or only a subset?
- if NO revise or drop; if YES retain
- if NO, drop question; if YES, should one or both remain?
- if NO revise or drop; if YES retain
- if NO drop; if YES retain
- If NOT make it right; if YES retain
- if NO drop; if YES, retain
- If NO retain; if YES retain only if other info can be obtained
- if ALL retain; if only SUBSET retain if identifiable beforehand
