Survey errors +++ Flashcards
Respondents thought they knew the answer but they really didn’t.
Response bias
We got the results from our phone survey. We got a lot more females in their 30s and 40s than are in the actual population.
Non-response bias
The Coke example/ the language online example –> loved the website but people didn’t want to learn languages online. Researched the wrong thing/ asked the wrong questions
Surrogate error
A survey asks “how much would u pay for this product” but also didn’t leave enough space for people to answer
Instrument error
I bought a list of names of software engineers to send my survey out in the Silicon Valley area, but a lot of emails on the list were bad because many of the engineers had moved to other companies.
Frame error
Hired someone with poor computer skills who did a bad job entering the data (made mistakes typing)
Processing (input) error
Polled the wealthiest in the area to determine the demand for weekly cleaning. Turned out they all had a full time maid. Didn’t define my population well.
Population error
The person I hired to conduct the survey was rude and in a hurry.
Interviewer error
Even though I told the staff in the company to get an even mix of ages and genders to complete the survey they ended up getting more old women cause that was the easiest.
Selection error
Sample design errors
- selection error
- frame error
- population error
Measurement error
- processing (input) error
- response bias
- non-response bias
- interviewer bias/error
- instrument bias/error
- surrogate information error
A market research experiment:
The researcher changes an explanatory, independent marketing variable (e.g. the marketing mix) to observe changes in the dependent variable.
Causal research
The only type of research that can demonstrate that a change in one variable causes a predictable change in another variable
Laboratory experiments
Conducted in a controlled setting
Lab experiments- advantages
- Ability to control all variables
- Greater internal validity
Lab experiments- disadvantages
- May not have good external validity
- Not transferrable to the actual marketplace
Field experiments
Tests conducted outside the laboratory
Field experiments- advantages
- Creates realism of the environment
Field experiments- disadvantages
- Internal validity
- No control over extraneous factors
– Weather, temperature, humidity
– Light, power, Internet speed
– People present, influences on test subject
– Access to information
Validity
Degree to which an experiment actually measures
what it is trying to measure
Internal validity
The extent to which competing explanations can be ruled out.
External validity
The extent to which causal relationships can be generalized to outside persons, settings, and times
Lab experiments
- High internal validity
- Low external validity
Will the results be the same in the real world though?
Field experiments
- Low internal validity
- High external validity
Experimental design
- control group (experimental vs control group) –> random selection
- A/B test for completely new product
Extraneous variables- threats to experimental validity
– History- intervention between the beginning and end of an experiments
– Maturation- changes in subjects occurring during the experiment
– Instrument Variation- changes in measurement instruments
– Selection Bias- systematic differences between the test group and the control group
* Randomization or matching
– Mortality- loss of test units or subjects during the course of an experiment
– Testing Effect - effect that is a by-product of the research process itself (prior exposure, e.g., to product, ad, etc.) – Regression to the Mean (see Hawthorne effect*)
Observation research definition
Systematic recording of behaviors, mostly without communicating with people involved.
Approaches to observation research:
– Natural Versus Contrived Situations
– Open Versus Disguised Observation
– Structured Versus Unstructured
– Human Versus Machine Observers
– Direct Versus Indirect
Advantages of observations over surveys:
– See what people actually do + physical reactions
– Avoids survey bias (e.g., interviewer, response)
– Quick, accurate data collection (with technology)
Disadvantages of observation vs surveys
– Researcher does not learn motives
– Cannot determine future behavior
– Time-consuming and expensive (by humans)
Four variations of a mystery shopper:
– Level 1 - a simple phone call for basic info
– Level 2 - a quick purchase, little or no interaction
– Level3-scripted conversation: assess basic knowledge, product options
– Level4– in-depth assessment of technical knowledge of product solution
Dependent vs independent variables (experimental vs treatment)
Treatment variable: Independent variable that is manipulated in an experiment
Experimental variable: The dependent variable that is measured
Four elements included in an experimental design:
- The treatment, or experimental, variable (independent variable) that is manipulated
- The subjects who participate in the experiment
- Adependent variable that ismeasured
- Some plan or procedure for dealing with extraneous causal factors
Traditional marketing vs big data
Traditional:
Traditional marketing research wants to learn about how people feel about a product, a company, or an ad. Researchers want to better understand what people were thinking when they made or didn’t make a purchase decision. Marketing researchers hope that by having a better understanding of a person’s feelings and how they think, then this will lead to a better comprehension of “why.”
Big data:
Big data and marketing analytics take a different tack. Ifthese new tools can accu- rately predict what people will do, then “why” isn’t really important. As you can imagine, this has led to conflict in some organizations between traditional marketing researchers and persons working in marketing analytics using big data.
Discussion guide
A discussion guide is a written outline of the topics to be covered during the session. Usually, the moderator gener- ates the guide based on research objectives and client information needs.
Projective tests
Technique for tapping respondents’ deepest feelings by having them project those feelings into an unstructured situation.
The rationale behind projective tests comes from the knowledge that people are often reluctant or unable to reveal their deepest feelings. In some instances, they are unaware of those feelings because of psychological defense mechanisms.
- word association test
- analogies
- sentence and story completion tests
- cartoon tests
- story telling
- third person technique
