Lecture 7 - Survey design Flashcards
(!) Describe surveys in general & when to use it
General:
- Produce systematic datasets
- Find cause &pattern by comparison
- Cross-sectional, time-series or panel/ multilevel research design
- Naturally-occurring variance
- If complex: Poss. Combine w. secondary data as proxy
When to use it:
- Subjective attitudes, values, intention & opinions
- Social desirability bias
- Sensitive topics
- Existing understandability
- If not lacking encoding: Unaware or forgot
- If no omissions or intrusions: Confused when actually occurred
- Able & willing to answer
- Data obtainable by structured questions
- Expectation of reliable answers
- Knowledge of how to use answers
- High expected response rate
- Capturing external phenomena
- Needed info for answering RQ
(!) Describe advantages & problems of using a survey
Advantages:
- Data from direct source
- Cheap
- Efficient
- Sample represent population
- Large sample
- High response rate
- Possible generalization
- High external validity
Problems:
- Recall bias
- Reported > Actual behavior
- Self-reported data
- Low internal validity
- Question refused answered
- Answer in middle of scale
- Participant stop
- Completion time
- Superficial measures: Dansk: Overfladiske
- Standardized instrument
- No poss. causality: Low internal validity
(!) Describe the different types of variables
Continuous:
- Interval: Number
- Ratio: Percentage
- DV for ANOVA & regression analysis
Categorical:
- Ordinal
- Likert scale
- Can be averaged to sum scale
- Should also be reversed
Nominal:
- Groups
Describe the stages in surveys
Design:
- Study
- Sampling
- Timing
Delivery
- Quality
Construct survey-testing:
- Literature review before
- Min. 5 times trial-measure response time
Collect data:
- Two reminders
- 2 weeks & 1 month
Analyse & present results:
- Every section has meaning
(!) Describe different quantitative sampling methods
Probability sampling methods:
General:
- Create representativeness for generalizing
- Large sample not always representative: Some never participate
Methods:
- Simple random sampling
- Systematic: Every 3. person
- Stratified: Ensure interest group well represented in sample
- Multi-stage cluster: Smaller & smaler group at each stage
_________
Non-probability sampling methods:
General:
- Describe/explore phenomena
Methods:
- Purposive samling
- Quota sampling: Group w. certain characteristis
- Availability samples-testing survey: Exploratory research. Low requirement for N
(!) Describe the considerations on response rates & how to ensure them online
Considerations:
- Lower if survey
- Important since “no response bias”
- Important since small samples can be biased
- Can be problem in sample: Not if random
- Higher in Scandinavia
Solutions:
- Quality of first contact: Explain purpose & credibility
- Simple design
- Good filter questions: Eliminate people
- Possibility to skip question
- Define terms
- Confidentiality: Tell its anonymous
- Short time
- Send reminder
- Handle by weights
(!) Describe the pros and cons of interviews vs. self-administered
Interview:
General:
- In person/telephone
Pros:
- Can clarify question
- Encourage participation/completion
- Higher response-rate
- Allow feedback & random question order
Cons:
- Bias-interviewer effects
- Less anonymous
___________
Self-administered
General:
- Postal/online:
Pros:
- Easy access
- Efficient
- Low cost
- Fast
- Anonymous
Cons:
- Low response rates
- Low respondent involvement
- Ambiguities not detected- false responses
Describe different questions & response types
Question types:
- Reversed questions
- Introducing questions: “Tell me about..”
- Follow-up questions: “What do you mean..”. Succes in analytical interview
- Probing questions: “Can you extend..”
- Specifying questions: “What effect did..”
- Direct questions: “Are you happy with ..”
- Indirect questions: “What do fellow students think of..”
- Structuring questions: “Lets move on to..”
- Silence: Pause for thought: Moment to reflect
- Interpreting questions: “Is it correct to say that..”
Response types:
- Yes/No
- Close/open response
- Frequency
- Ratings/ranking
- Agree/Disagree
- Bipolar/unipolar measure
(!) Describe quantitative biases & solutions to them
No response bias:
- Problem if hard-to-reach group despite large sample
- Problem if specific groups
__________
Social desirability bias:
- Over-report admirable attitudes & behavior
- Solution: Ask indirectly. Rephrase
__________
Satisficing:
General:
- Answer survey as fast as possible
- Order effects: Potential spill-over effect due to chronology
Causes:
- Task difficulty
- Ability: Maybe tired
- Motivation
Solutions:
- Clear questions
- Simple terms
- Short survey
- Show few questions at same time
__________
Acquiescence:
General:
- Tendency to agree with any content
- If low cognitive ability, high fatique & guess encouragement
- Increased by satisficing
- Create common method variance bias
Solutions:
- No agree/disagree option
- Ask more than once: Cross check
- Reversed questions
__________
Common method variance bias /CMV:
General:
- Covariance due to same method
- Fake correlation of variables: Made by researcher
Main sources:
- Same rater
- Response style
- Item wording
- Item context
Solutions:
- IV & DV from different sources
- Separate variables: Cover story prevent understood expected correlation
- Eliminate common scale
- Improve scale items: Both unipolar & bipolar
- Reduce social desirability bias
- Balance positive/negative: Harder to rush through
- Create fake CMV: Statistical remedy
__________
Conversation conventions:
- More interpretation on later vs. earlier in text
- Eg. Listen when others are talking
- Solution: Always provide new info
- Ref. Analytical interview
(?) Describe important considerations about questions when events, behavior and attitudes and subjective phenomena attitudes
Events, behavior & attitudes:
General:
- Often in context
- Lack encoding: Not being aware or remember
- Omissions or intrusions: Confused when it actually occurred
- Either define event on beforehand or free-writing text
- Awareness of sensitive topics
Potential problems:
- Hard recall
- Change in memory
- Questions influence interpretation
Provide visual help:
- Give calendar er some key events: Eg. Holiday or news
- Define reference point before question
- Bounded interview/bounded recall
___________
Subjective phenomena attitudes:
- Differ since no objective truth
- Bigger risk of priming format/design: Social desirability
- Object & evaluative dimensions: Approval or disapproval
(!) Describe considerations on scales
General:
- Often Likert
- Best to use 7-9 points
- Turn into sum scale
- More than numbers: Eg. “Strongly agree”
- Numbers should match words: Eg. -5/+5 vs. 0-10 perceived diff.
__________
Bipolar vs. unipolar:
General:
- Using both reduce satisficing
Bipolar:
- Neutral midpoint
- Extreme negative & positive ends
- Incl. Opposite
Unipolar:
- From 0 - Extreme
- Excl. Opposite
(!) Describe different considerations on questions
General:
- Both for surveys & interviews
- Remember diff. understanding
- Question influence answer
Do:
- Relate to RQ
- Clear question & one at time
- Use participant language
- Facilitate lack awareness: Reference point
- Capture interviewees perspective
- Clarify answers: Why, how, who, when, what
- Facilitate lack of memory: Card, picture, specific word, walk through
- End on good note: Not sensitive topic
- Chronology from easy to hard questions
- Info about interviewee: Age, Gender etc.
- Guide response: Bring back to question
- Ask examples
- Mix diff. questions
Dont’s
- Leading questions
- Ambiguous questions
- Double questions
- Academic or technical terms: Eg. Unpack “motivation” term
- Vague terms
- Close-ended question
- The RQ: Researcher job
- Non-important or available question
