Surveys 6.2 & 6.3 Flashcards
In-person advantages
- ppts can ask for clarification
* relatively high response rate
In-person disadvantages
- relatively more expensive
- takes longer to collect the data
- potential for interviewer bias
- Use trained interviewers
- increased risk of socially desirable responses
- Convince ppts data will be anonymous and use trained interviewers
- Sampling bias – respondents not representative of target population
Self-administered surveys advantages
- Relatively inexpensive
* Easy to distribute
Self-administered disadvantages
- Lower response rate
- Sampling bias
- Make the survey as accessible as possible (e.g., email, social media…)
- Ppts may not be who they claim to be
- Ppts can’t ask for clarification
- May still get socially desirable responses but less likely than in-person
- No interviewer bias but still need to worry about biased questions
Practices for reducing bias
•Use trained interviewers •Make survey anonymous •Make the survey easy to access, understand and complete •Easily available to those representative of your population •Not easy to access for those who are NOT members of your target population •Overall format and structure •Questions are clearly written and unbiased
Satisficing
doing the minimum to complete a task
•For surveys, the minimum is “putting an answer”
Speed runs
Rushing through without paying attention
Straight lining
Giving identical answers on each question
•E.g., always selecting “strongly agree”
Addressing satisficing
•Think carefully about the incentives that are offered •Don’t make the survey too long •Include catch trials •Select A on this question •Ask similar questions •Check for consistency (internal reliability) •Switch the polarity of the questions •Strongly agree to Strongly disagree (I like cats I dislike kittens) •Switch the polarity of the anchors •Be careful, people may not notice or may get confused
Response burden
The effort required for a participant to complete a survey •The greater the burden, the less likely people will complete it •The less likely they will return for follow-ups
Layout of Survey Questions
The survey should appear relatively
easy and interesting
Topics should be organized so that they
are easy to follow and motivate ppts to
keep going
Structural organization
Goal: get participants to complete the survey and give you good data
Funnel structure:
Beginning
•broad, general interest questions that are easy to answer
•warm the ppt up and get them involved in the survey
Middle
•Questions with a narrower focus; and those that are more difficult, sensitive, or time
consuming
•Ppts have invested some time and they aren’t burnt out
End
•Demographic questions
•Easy to answer, important, but not all that interesting
•Also answering these questions can induce stereotype threat which could bias later responses
Branching using filter questions
When how you
respond on one question
influences the questions you will
see next. Useful to avoid redundant or irrelevant questions.
Filter questions: questions that lead to branching
Battery
•A set of questions that share an introduction and that all have the same response categories •Conceptual alignment makes it easier to think through and take
Open-ended questions
No structure provided for the answer
Benefits: •Ppts can include lots of details •Ppts can include explanations •Ppts may include info you had not thought to ask about
Drawbacks
•Coding the data
•Time consuming, difficult
Closed-response questions
•Pre-established response options
Benefits:
•They are already coded
•Easy to analyze
Drawbacks: •Potential for an incomplete set of alternative responses •no chance for ppts to explain choices •may lead to less info than what could be gained by using open-ended Q’s
Writing good survey questions
•Brief – keep it brief and to the point
•Relevant – only ask about topics relevant to the research
•Unambiguous – readers should interpret it the same way
•Specific – each question should ask only one thing
•Objective – the questions should not be written in a way that biases
the readers to respond one way or another
Loaded questions
contain an unjustified assumption
Leading questions
lead subjects to respond with a certain answer
Double-baralled questions
ask more than one question at a time
Considerations
- Use appropriate language
- Avoid double-negatives
- Be careful with extreme quantifiers (all, none, every, etc.)
After survey
- Get expert feedback
- Get user feedback:Pilot it
Interobserver reliability
•How reliable are your measurements?
•Dealing with disagreements
Resolve them using a predetermined method
•Interobserver reliability for two coders
Proportion agreement (number of agreed scores / total scores) •Limitation: Doesn’t consider the possibility that agreement occurred by chance
Cohen’s Kappa (better and more common)
•Accounts for chance agreements to provide a better metric of reliability
•For categorical data
