LEC 7b Qualitative Survey Flashcards
Surveys
Web-based
Mail
interviews:
Telephone
Personal
Focus group discussions
Delphi technique
1851 - 1864 (Henry Mayhew)
4 volume London Labour and London Poor
Based on conversations and observations of daily life of street people
1906 (B. Sebohm Rownstree)
A study of town life
Examined the extent of urban poverty
Several forces re-shaped the social survey into modern quantitative survey research:
Researchers
Post WWII
Data storage
Funding
Methodology
Historical Overview
From the 1890’s to 1930’s survey became the major method for social research
It became part of an important part of the “Social Survey Movement” in Canada, US, and Britain* In the beginning the surveys were largely exploratory and descriptive
Used a a means of informing the public of the problems related to industrialism and to provide info for democratic decision making
Researchers…
-Applied statistically based sampling techniques and precise measurement
-Adapted the survey to applied areas
-Social research was re-oriented away from non-academics towards “scientific” methods modeled after the social sciences were implemented
Post WWII
During the war, the governments used survey research extensively
However, when many researchers who had worked for the government during the war returned to the universities, the universities were reluctant to use survey research
universities were reluctant to use survey research because of
-Cost
-Number of subjects required
-Sceptical of using something that had evolved from “private industry”
1970’s
Use of surveys continued to grow
Significantly greater acceptance with academia
Communication, education, economics, political science, social psychology, sociology
Factors contributing to growth in survey research post 1970’s:
Advent of computers
Evolution of social research centres with expertise and interest in quantitative research within university settings (n = approximately 50)
Data storage
Data archives were created to store and permit sharing of large scale survey data for secondary analyses
Funding
Federal governments have expanded funding for research and development in the social sciences dramatically between 1970’s and 1980’s
Methodology
-Substantial research was conducted to improve the validity of surveys
-Errors in previous methodology were identified and corrected
-Created improved statistics for analyzing data
Modern Day Surveys
Using a sample of many respondents, surveys ask many questions, measure many variables, test multiple hypothesis, and can infer temporal order from questions about past behavior, experiences or characteristics
Web based surveys
3 types
Email
USB key
Forms based
Web based surveys: Email
-Better than regular mail in terms of dissemination and return
-Very large number of respondents
Web based surveys: USB
Questions are loaded onto a USB and sent to respondent
Web based surveys: Forms based
Questionnaire is located on the researches site
Mail Survey: Advantages
- Cost surveys (especially compared to interview technique)
- No interviewer bias
- Greater assurance of anonymity
- Competition by the respondent at their convenience
- Accessibility to a wide geographical region
- Accurate info (resp, can check records)
- Identical wording for all respondents
Mail Survey: Disadvantages
- Lack of flexibility
- Likelihood of unanswered
- For response rate (compared to interviews)
- Inability to record spontaneous reactions and or nonverbal responses
- Lack of control over the order in which questions are answered, and over the immediate environment
- No guarantee of return by deadline date
- Inability to use a complex questionnaire format
Survey Design: Descriptive designs
Emphasise what characteristics the group or groups posses
Will usually use cross sectional, longitudinal, or group comparison
Difference amount the tree is the time factor (note; group comparison does not necessarily have a “time” factor)
Survey Design: Analytical designs
Attempt to explore/explain “why” certain groups have certain characteristics
Attempt to address the relationship between variables
Cross sectional, case control, prospective
Cross sectional= cases from the whole population
Case control= one for one
Prospective= looking ahead
Question Design: Closed ended questions
Fixed answers (eg. yes/no;/male/female)
Closed ended questions—Advantages
Easy to complete
Simplifies coding and analysis
Better response rate for sensitive questions
Minimum of “extra” comments
Closed ended questions—Disadvantages
Unsure of question may result in guess or random selection
Variations in responses is reduced because of restrictiveness of lists
Too many answer categories
Increased levels of frustration
Incorrect answer selection (errors)
Question Design: Open Ended Questions
Aka unrestricted, free range answers
Open Ended Questions—Advantages
-Usable when all answers categories are unknown
-Preferred for controversial, sensitive or complex issues
-Allows for creativity, clarification and detail
Open Ended Questions—Disadvantages
-Difficult to code/ analyse
-Greater demands on the respondent( times, writing ability, thought, other?)
-Questions may be too general
-Data collected may be what the researchers was looking for
Avoid: Jargon
Words specific to a profession or group
Avoid: Slang
Jargon within a subculture
Avoid: Abbreviations
Sx, CPX, Hx
Avoid: Ambiguity
Eg, what is your income?
Avoid: Confusion
Avoid: Vagueness
Do you workout regularly ?
Avoid Emotional language
Eg. How do you feel about murderous mothers?
Avoid Prestige Bias
Designing the question such that a highly respected group or individual is associated with an answer choice
Eg. Most doctors say that cigarette smoking causes lung disease for those near a smoker- do you agree?
Avoid adjectives without agrees upon meaning
Eg. “Steeped”- tim hortons commercial
Avoid double barreled questions
A question that contains more than one issue and can create respondent confusion or ambiguous answers
Eg. do you exercise or play sports regularly?
Avoid double negatives
grammatically incorrect and very confusing
e.g., I do not not like you. (?so do you likeme or not?)
Avoid asking about distant future events
Eg. suppose a new grocery store opened down the road- would you shop there?
Avoid leading questions
Aka as loaded questions
All responses should be considered legitimate
Eg. you don’t smoke, do you?
Avoid asking questions that are beyond the respondents capabilities
Will lead to frustration, poor quality of answers, and poor answer rates
Eg. Do you feel that multivariate regression is a better statistical approach for studying differences than logistic regression?
Avoid false premises
Do not begin a question with a premise the respondents may not agree with, then ask them to make choices about it e.g., The post office is open too many hours.
Do you want it to be open for four hours later or close four hours earlier each day?
Avoid overlapping or unbalanced response categories
choices should be mutually exclusive
e.g., 10 – 15, 15-20, 20 - 25
choices should be exhaustive
e.g., years of schooling – high school, some college or some university, university, college, graduate, post-graduate, post-doctoral, fellowship
categories should be balanced
e.g., 10 – 15, 16 – 20, 21 – 42, 43 - 48
Question should
-use clearly worded items
-short questions KISS principle
-only one idea (concept) per question
-overall length Goldilocks not to long not to short
if your survey has close-ended questions (e.g., you are collecting numbers / numerical information), your research design is
quantitative
if your survey has open-ended questions (e.g., you are collecting words), your research design is
qualitative
regardless of whether your research design is quantitative or qualitative, the people completing your survey are
respondents
—>they are not subjects and they are not participants
Scales definition
a class of quantitative data measures often used in survey research that captures the intensity, direction, level, or potency of a variable construct along a continuum
most scales are at the ordinal level of measurement
Scales most common purpose
When a researcher wants to measure how an individual feels or thinks about somethin
Can produce quantitative measurements to test hypothesis
Scales assume that people with the same subjective feeling mark the graphic scale at the same place
Numerical rating scales
Pain scales; 0= no pain, 10= worst pain ever
“Rate your pain level”
Checklist rating scales
Given a list of items
Most often the question will say “check all that apply”
Forced choice rating scale
Given two or three options to choose from
Rankings Rating Scale
Rank the items frim most favorite to least favorite
Types of rating scales
Numerical
Checklist
Forced choice
Rankings
Likert Scales
Developed in 1930s by Rennis Likert
Used to provide an ordinal- level measure of people’s attitudes
Also known as summated- rating scale, or, additive scale because a person’s score can be computed by summing the number of responses a person gives
Usually ask people to indicate whether they agree or disagree
Likert Scales Definition
-a scale most often used in survey research in which people express attitudes or other responses in terms of ordinal level categories (e.g., agree / disagree) that are ranked along a
continuum.
-A likert scale is ordinal NOT interval
Numbers are used for convenience
minimum number categories = 2
maximum number of categories = 8
can collapse categories if needed – but you CANNOT expand categories once they have been made
Likert Scales: Response Set
—-a tendency to agree with every question in a series rather than carefully think through one’s answer to each
Aka response style or response bias
To Avoid: word statements in alternative directions
answers will either be inconsistent, or, will appear as having a contradictory opinion
Likert Scales: Indexing Responses
- assigning scores to answers in order to give a
more precise quantitative measure of a person’s opinion.
Example: scoring for each item: 1 = disagree, 2 = don’t know, 3 = agree
-maximum score will depend on the number of items on the scale, but it would be equivalent to adding up all the “agree”
-minimum score = as above except now you are adding the “disagree” items
Semantic Differential Scale
- Developed in the 1950’s
- Used to provide an indirect measure of how a person feels about a concept, object, or other person
- Measures subjective feelings using adjectives
- Most adjectives have polar opposites (eg. light/ dark)
Paired opposite adjectives with a continuum of 7-11 points between them (?why odd number)
Subject marks the spot on the continuum between the adjectives that expresses their feelings
Anchors- positive on one end, negative on the other
How do you feel about winter?
Happy __ ___ ___ ___ ___ __ sad
Semantic Differential Scale: Three major classes of meaning:
Evaluation (eg. good-bad)
Polency (eg. strong-weak)
Activity (eg. Active- passive)
Results from the scale can tell a researcher how one person perceives different concepts or how different people view the same concept
Errors Made on Rating Scales
Rating error:
Ieiency
Central Tendency
Halo
Proximity Errors:
Observer bias
Observer expectation
Rating Error: Ieiency
Too easy going; tend to be very generous
E.g., peer evaluations
Rating Error: Central Tendency
Person completing the scale tends to stay toward the middle; avoids extremes of the scale
May be leaving “room for improvement”
Rating Error: Halo
Previous impressions or knowledge of a person or subject will impact all the answers
The halo “surrounds” the responses
Proximity Errors
When the rater considers certain behaviours to be nearly the same when they are listed close together on a scale than when they are separated by some distance
E.g., “active” and “friendly”
Becomes more of a problem when the rater may not understand all the words on the list
E.g., salacious
Proximity error: Observer bias
Rater has their own biases and prejudices that affect how they answer the question
racial , sexual and philosophical biases are more prominent
Proximity error: Observer expectation
Provide answers that reflect what the rater “expects” the answer should be
Stereotypes
Factors Affecting Mail Surveys
Sponsorship → ?legitimate organisation
Questionnaire colour → irrelevant
Length → no fluff, only what you want
Easy, easy, easy
How do I get back to you?
Who did you mail this to?
First class versus business class
Lick-and-stick stamps versus business stamp
Time of mailing
Weekday patterns
February and april- ick
March → good
September → school only
Incentives → $$$ always a good thing
Follow-up letters / telephone calls
Rates of return
Eligibility Rate
Location Rate
Response Rate
Contact Rate
Non- Contact Rate
Non response Rate
Rates of Return: Eligibility Rate
Percentage of contacted respondents who are eligible
Rates of Return: Location Rate
Percentage of people in the sampling frame who are located
Rates of Return: Response Rate
Response rates / total response rate
Number of surveys completed given the number of eligible people to compete
Rates of Return: Contact Rate
Percentage of contacted, eligible respondents who agree to participate
Rates of Return: Non- Contact Rate
- Important when trying to identify how many people could not have been contacted (e.g., isolated areas)
Rates of Return: Non response Rate
Aka refusal rate
Number of surveys not completed
90% is amazing → very unusual
50-60% → preferred / desirable
30% → happy
Saying nothing speaks volumes
Steps in Developing, Designing and Implementing a Survey
- Plan the survey
- Overall design
- Methods of data collection
- Drawing the sample
- Questionnaire construction
- Administering the questionnaire
- Data entry
- Tabulation
- Analysis
- Recording and reporting
Interviews
Telephone interview
Personal interview
Focus group discussions
Delphi Technique
Telephone Interview—Advantages:
Cost saving compared to face to face interview
Faster than mail or personal interview
Can cover broader geographical area
Increased degree of anonymity
Monitoring and quality control
Call backs if no answer
Accessibility / safety
Better for collection of sensitive information\
Telephone Interview—Disadvantages
Less motivated to respond
?hoax?cover up
Use of checklists / visual aids → N/A
Little control over situation
Personal Interview
Sit down meeting between one respondent (the interviewee) and the researcher (interviewer)
Personal Interview—advantages:
-Personalization of the study to the participant
Flexibility (allows for further probing)
Higher response rate
Observation of verbal and non-verbal responses
Control over question order
Spontaneity- no additional help from others
Recording of the length of time to complete
Ability to use more involved and complex questions
Personal Interview—disadvantages
Cost in terms of money and time
Openness to manipulation
Vulnerability to personality clashes
Lack of anonymity
Inconvenience / ?record checking
Lack of standardisation in questions because of probing; possibility of re-wording or re-phrasing can change meaning
Lack of access to certain respondents (eg. geographically isolated)
Difficult to summarise findings
Personal Interview Structure: Unstructured
Broad freedom in terms of time and responses
Usually used for very personal information
Personal Interview Structure: Semi- Structured
Core set of questions
Interviewer can move in related directions for in-depth probing
Personal Interview Structure: Structured
Well-defined pattern of questions (similar to a questionnaire)
Can only clarify or elaborate on answers
Interview Bias:
Personal opinions overshadow the respondents’ answers
Interview bias: Errors by the respondent
Forgetting, embarrassing, misunderstanding, lying
May due to the presence of others
Interview Bias: Unintentional errors (aka interviewer sloppiness)
Contacting the wrong person
Misreading the question
Reading questions in the wrong order
Recording the wrong answer
Interview bias: Intentional subversion by the interviewer
Purposeful alteration of the answers
Omission or rewording
Altering questions or choice of respondent
Influence due to the interviewer’s expectations about the respondents’ answers based on appearance, living situation, etc.
Failure to probe, or to probe properly
Delphi Technique
Used in settings to achieve a group consensus
Each individual provides feedback
Re-consider your position in light of other comments provided
Use of multiple questionnaires in a temporal order to reach final goal of consensus
When Asking Questions Should Avoid
Jargon
Slang
Abrreviations
Ambiguity
vagueness
confusion
