“Conducting Experiments for Gaining Customer Insights aka data marketing experiments Flashcards
casual conclusion
how one factor caused a change in another variable of interest
casual research
aka experiments
allows a firm to make cause and effect statement
cause and effect stamenent
can be made if it can be shown with a high level of internal validity that the casual variable or indepedent variable was responsible for the change in the outcome variable or dependent variable
objective of experimental design
to create groups of respondents in a way that observed changes in one group can only be attributed to the casual variable
done through experimental design
The main goal of experimental design is to figure out if one thing (the cause) is really affecting something else (the effect). To do this, we create different groups of people (respondents) and treat them differently in a way that we can clearly see if the changes in one group are caused by the factor we’re testing
experimental design
carefully gather data and then evaluates it to draw conclusions then do statistical analysis
when looking at causality, use a control and trewtaenet group
in the treatment group you vary the casual factor aka the indepedent factor
then observe the level of dependent variable
three conditions allow us to infer casualty
- if the two groups are equivalent in all other respects minus what u are testing
- of there is a stisatical association
- we can rule out other possible explanations for the observed association
if yes we can conclude that the independent variable causes a change in the dependent variable
customer focused experiments should balance two types of validity
internal validity and external
internal validity
an experiments ability to unambiguously show a cause and effect relationship –> the extent to which we can attribute the change in dependent variable to the indepdent variable and not to other factor
major concern for internal validity
have to ensure that the treatment group and control group are equal
achieved through random assignment or matched samples
random assignment
occurs when each experimental unit (ex: each salesperson) is randomly assigned for inclusion in the control and treatment group
each salesperson has an equally likely chance of being assigned to either the treatment or control group
statistically compare each group
use this but if not possible, use matched samples
if treatment and control are unequal
we have selection bias – when the treatment group is chosen using a methodology that precludes random assignment
random assignment is not always feasible however
matched sample
as a technique for generating a control group is used when legal, ethical and practical consideration preclude random assignment
external validity
the extent to which results of the experiment can be generalized or extrasploted to oder people, settings and time
do the results from one experiment with high internal validity apply to other situations
differences in setting can limit external validity
systemTIC BAISES
can threaten the random assignment protocol
threats to internal validity
- history–> any variable or events other than the ones in the xperiemnet that affect the value of the Fdepdenet variable
Events or variables outside the experiment that affect the dependent variable during the course of the experiment.
Example: If you’re testing the effect of a new advertisement on sales, but during the experiment, a major economic event (like a sudden price increase for competitors) occurs, this could influence the sales, making it hard to tell if the ad itself caused the sales changes
- maturation–> any economic, biological or psychological processes that systematical vary with passage of time
Changes within participants over time that are not due to the experimental treatment but to natural processes (e.g., aging, learning, fatigue).
Example: In a long-term study on the impact of a new training program, employees might improve over time due to gaining experience or becoming more familiar with their tasks, not because of the training itself
- instrument varaiton–> a change in the measuring instrument t
- experimental unti moratloty–> differential loss of respondents from dif groups
- pre messuremnet/ interactive testing-
- selection bias
- statically regression –> special type of selection bias
- enchahing effect of selection bias –> non random assignment
external validity threats
- surrogate situation–> when the expeiemtal task is so contrived that it is unlikely to be a reasonable proxy of the reality that the researcher wants to study
- measurement of dependent variable: whe deponent variable does not map on to the behaviour or construct of interest
- meaduremtn timing- If the timing of measurements in the study is not optimal or does not align with when the effects would occur in the real world, the findings might lack relevance when applied outside the study.
- sample selection –> sample may not be representative of the population
internal vs external. which is more desirable?
internal is necessary for drawing casual conclusions so even if it has high external not good
customer focused firms should strive to optimize both
use high internal to draw conclusions but use external to go beyond the conlsisions
there is a trade off between external ad internal for single customer focused research studies
pseudo (bad) designs for customer focused expriemets
many lack internal validity –. cant establish causality
1. one shot cause study –> lacks a control group that is equivalent to the treatment
2. one group pre/ post test design–> faulty pre post
3. static group comparison
- show that there is a need for a control group that is the same as the tatrement and best accomplished through rands assignment
true (good) designs for customer focused exorieennts
- post test only, control group design–> rather than creating the tatrement group based on judgement or cmveneimce, experimental units are randomly assigned to the treatment or control group
- pre test/ post test control group design–> before, exposing he treatment group to the treatment, but after random asisgmemt, a pre measurement is taken on the deponent variable. serves as the baseline from which changes in the deponent variable can be evaluated
- avdnavced casual experiments
- time series or quasi experiments
overall
- establish two equivalent groups aka control and treatment , established through random assignment or if need be matching
- researcher has to carefully evaluate the study and its context to asses threats to external and indternal. internal - high, casual inference can be made. external = low, care should be taken o project the results beyond the context of the current experiment
- conducting the approbate statistical analysis should answer 3 questions: iOS the difference statistically sigfniacant, how large is the impact, os the impact pos or neg. we use ANOVA
- researcher should assess the applicably of the results. just bc results are statistically significant may not mean they are relevant, useful of applicable
three types of marketing research
exploratory, descriptive, casual
(eiley digs Chloe)
exploratory
methods: case studies, focus groups
felxible, versatile , discobery of ideas and insights
descriptive
surveys, panels
preplanned, structured, conclusive
describe market characteristics or functions
casual
experiments
manipulation and control of variables
determine cause and effect relationships
focus group example
New smart HDTV focus group
May have a focus group where questions will be asked in an open ended manner
Show a prototype tv in action
Ask open ended questions: what do you think, like, dislike, change, questions
Let participants speak freely
Content analysis of focus groups discussion→ Content analysis” of focus group discussion => analysis in the form of words (not numbers) => e.g., verbatim statements made by particiipants (maybe frequency of mention can be a numerical output)
Identify important features & other (emergent) issues not known to marketer yet
Before spending a large budget on launching a compagne may want to show to focus group to ask their likes and dislikes
Different book covers may be tested in focus group
Look at focus group answers , show them something and ask them questions
Goal: To understand participants’ initial impressions, likes, dislikes, and suggested improvements regarding the new smart HDTV, as well as uncover any unknown needs or preferences.
Participants: Select a sample of diverse consumers representing your target market, ensuring they can provide a range of perspectives.
Prototype: Show a working prototype of the smart HDTV to the participants, allowing them to interact with it directly to experience its features.
quantitive
Measured in numbers
Two examples : customer satisfaction surveys (we want to actually measure customer satisfaction and benchmark it against competitors )
Attribute Importance Survey - segmentation of customers
Can do surveys to understand how to position your brand to different segments
What Are the important attributes in each segment
Can do a survey and give a list of attributes to be ranked in order of importance
experiment
A survey: has questions and numeral answers
Only experiments can determine cause and effect
cause and effect: very important concept, cause and effect is important because marketers want change something in the marketing mix (cause) the response is the effect
For example, suppose a company adjusts its product pricing, ad campaign, or placement strategy (all parts of the marketing mix). In that case, they want to know if these changes directly lead to a desired outcome, like increased sales, more customer engagement, or higher brand loyalty
If case and effect is so important, need to know that a certain cause leads to a certain effect
Four conditions to be sure
to establish a cause and effect relationship
1) association –> need correlation between cause and effect
2) temporal –> cause must happen before the effect ( If you’re testing whether an exercise program causes weight loss, the exercise program needs to start before the weight loss happens, not after)
3) eliminating other causes
4) mechanism between cause and effect (There should be a logical or plausible mechanism that explains how the cause leads to the effect. Without this mechanism, it’s harder to justify that the cause is directly responsible for the outcome.) ex: vaccines and immunity. the mechanism is that it increases the number of white blood cells
alyson tries making eggs
temporal
Have to administer the cause first and then do the effect later bc cause leads to effect
Necessary but not sufficient bc next two are important
Things that have colrreation and temporal precedence but arent cause and effect
Ex: length of womens skirts based on fashion and stock market performance , there is a positive correlation between these 2, no underlying cause
No cause but happens to be correlated
Ex: correlation vs causality
Dont conclude causality from temporal and association alone
if a brand wants to claim that an ad campaign led to higher sales, the campaign has to run first, with sales increasing afterward. This sequencing—where the “cause” precedes the “effect”—is crucial but isn’t enough to establish causality by itself.
correlation isnt causation: For instance, the length of women’s skirts (a fashion trend) and stock market performance have shown a positive correlation in past studies, but it’s unlikely that one truly affects the other.
eliminating other causes
important
Apple day keeps the doctor away: apple is the cause, keeping the doctor away is the effect. You can ask a bunch of people if this is true.
Does this mean eating apples leads to better health? Not necessarily, may be other hidden variables or background variables (maybe none of them smoke or just lead healthier lives, dont know the other causes)
1) control vs experimental
2) measure other causes and use covariates
3)list other causes and eliminate them
POST TEST only control group
how to rule out other causes
Cause and effect in marketing: online experiment
If a certain add works better than another in an experiment, have to ask why? The logical reasons in marketing have to do with concepts→ our consumers are high effort, have attribute information they consider important, or my customers are low effort and respond well to heuristics (emotions)
Why does washing your hands kill germs? soap breaks down germ cells
Why did people keep believing this false cause and effect claim: authority norm, social proof, confirmation bias, correlation and temporal precedence
Correlation and temporal precedence: did bloodletting and patient improved, but logically they would have improved anyway without the blood letting (confirmation bias will continue to tell people blood letting works)
Cause and effect→ internal validity
External validity→ generalizability
You can genralize sometimes → three conditions to be met
External Validity:
Ability to generalize cause→ effect to a broader population
External validity refers to whether the cause adn effect relationship found in the experiment can be generalized tot he population in question (rather than only those who took part in the experiment)
High external validity (generalizable) if
using right sampling frame that covers entire population
random sampling of experiment participants from the population
Pick an adequate sample size ( 1000-2000)
Need two randoms
threats to external valdiidty
Surrogate situation ;
Experiment is so unrealsotsic that you cannot be confident that the results will apply real world
Measurement of dependent variable:
This happens when the way you measure the dependent variable (the outcome you’re studying) doesn’t accurately reflect the real-world behavior or concept of interest)
If you’re measuring customer satisfaction by only asking one yes/no question, you might miss nuances in how customers really feel, which could lead to inaccurate conclusions about satisfactio
Measurement timing;
(The time at which the dependent variable is measured can impact results. If the measurement occurs too early or too late, the results may not accurately capture the true effect of the treatment.
Example: If you’re testing a new training program, measuring employee performance immediately after training might not show long-term effects. Conversely, waiting too long might allow other factors to influence performance)
Sample selection ;
(This refers to whether the sample of participants in the experiment represents the larger population you want to generalize to. If the sample is not representative, the findings may not apply broadly)
Megan misses sams spit
cause and effect/ internal validity
design experiments with these basic principles in mind
- control group and experimental groups , random assignment of participants to conditions, measure dependent variable, measure poentntial covariates (measure the factors that could influence the outcome)
- do both pre pot measurement only if no pre measurement bias
- ensure cause comes before effect in time
- ensure there is a mechanism between cause and effect
= observe correlation between cause and effect
- then go through list of threats to cause and effect (i.e internal validity ) and logically rule them out
generalizability/ external validity
select participants with these basic principles in mind
- randomly select participants from the population using right samplig frame and right sample size
- then go trough the list of threats to geenralizatbikty (i.e external validity) and logically rule them out
what to do/ ask in a focus group
let partipcnats speak freely
show prototype
ask open ended questions
dont ask questions with numbers
association and temporal precedence
not enough to etsbalis cause and effect
pre test and post test vs just post test
Pre-Test and Post-Test Control Group (no anchoring bias) Design is ideal when you need to measure change over time and have more control over pre-existing group differences.
accurate but may have anchoring bias
Post-Test Only Control Group Design is simpler and avoids testing effects but requires careful randomization to ensure group equivalence.
simpler and less accurate
high external validity
- Using right sampling frame that covers entire population
– Random sampling of experiment participants from the population
– Pick an adequate sample size (1000-2000)
