reading 4

Question 1

Two main sets of methodologies is changing the way organizations do business in the current digital age:

Answer 1

big data analytics: data mining, machine learning and other statistical techniques allow practitioners to handle and analyze huge sets of data with a reasonable effort

Business field experiemnts: studies conducted outside of the lab by means of ease-to-use software allow managers to reliably answer causality questions at reasonable costs

Question 2

When should I consider setting up a field experiment to answer my research or business questions

Answer 2

field experiments are one of the most reliable ways to test a theory or to prove that a business action results in a desired outcome

Findings from field experiments have direct implications for business operations. In the language of experimentation, we say that they generalize well and have high external validity. On the other hand, lab experiments are acknowledged to have higher internal validity

Question 3

True experiments (either in the lab or the field) show three identifiable aspects

Answer 3

1) the responses of experimental subjects assigned to receive one treatment are compared to the responses of subjects assigned to another treatment (often a control gorup which receives some type of baseline treatment that is essentially no treatment or the state of the art condition. In the case of multivariate experiments, there are several treatment groups, which are all compared among each other

2) The assignment of subjects to each group is done through a randomization device such as a coin flip, a dice roll or algorithm
3) the manipulation of the treatment is under the control of an experimental researcher

Question 4

Natural experiments

Answer 4

share attribute 1 and partially atrtribute 2 of t rue experiments since assignment is random or as if random. However in such cases, data comes from naturally occurring phenomena and therefore the manipulation of treatment variables is not generally under the reserachers control

Question 5

Lab experiments

Answer 5

Participants are tested in an environment which is created by the researcher and which thus differs from reality. This unreal environment allows the experimenter to control other potential influences on the response but has the main drawback of making the respondent feel observed, which can lead to serveral kinds of response bias

Question 6

Field experiments

Answer 6

in which the setting is an everyday life situation, often the exact same setting where the findings from the experiment will be deployed. IN most field experiments, participants are not even conscious of taking part in an experiment eliminating the risk of incurring a response bias

Question 7

Key features of field experiments

Answer 7

authenticity of treatments, representativeness of participants, real world context, and relevant outcome measures

Question 8

Causal effect:

Answer 8

the difference between tow potential outcomes, one in which a subject receives the treatment and the other in whic hthe subject does not receive the treatment

Question 9

Experiments both in the lab and field provide unbiased estimates of the ATE when teh following assumptions are met:

Answer 9

1) random assignment: treatments are allocated such that all units have an equal probability between 0 and 1 of being assigned to the treatment group

2) excludability: the treatment must be defined clearly s otaht one can assess whether subjects are exposed to the intended treatment or to something else

3) noninterference: no matter which subjects the random assignment allocates to treatment or control, a given subjects potential outcomes remain the saem

Question 10

Internal validity:

Answer 10

refers tot he extent to which we can say the observed effect in our study was caused by our treatment

Question 11

Selection bias

Answer 11

when assignment to treatment is not random and certain types of people are more likely to receive one of the treatments, in other words, the experimental groups systematicall differ form each other

Question 12

Differentia lattrition:

Answer 12

when certain types of subjects drop out of one of the treatments. it implies that certain types of participants leave during the run of the experiment or do not take part in the final measurement

Question 13

Time effects

Answer 13

when treatments are administered at two different times, outside events, learningor other changes are confounded with the treatment

Question 14

Confounding variables:

Answer 14

when other vairables are correlated with the treatment and have an effect on the outcome, a cause effect relationship between the confounder and the dependent variable can be mistakenly assumed to be a causal effect of the treatment

Question 15

Noncompliance

Answer 15

subjects are assigned to the experiment do not get the specified treatment. THis can happend because of individuals voluntary decision to use a different treatment than the one they were assigned, because they do not like it or they think another treatment would be better

Question 16

Diffusion of the treatment across groups

Answer 16

subjects assigned to one treatment find out about the other treatment

Question 17

demand effects

Answer 17

participants guess the hypothesis of the experiment and try to cooperate by exhibiting behaviour that confirms the hypothesis

Question 18

Experimenter bias

Answer 18

makes subjective measurements and inadvertently favors the hypothesis in those measurements

Question 19

Hawthorne effect

Answer 19

it is also possible that individuals being part of an experiment and being monitored change their behaviour due to the attention they are receiving from researchers rather than because of the manipulation o fthe independent variables

Question 20

Ambiguous temporal precendence

Answer 20

in some experiments it can be unclear whether the treatment was administered before or after the effect was measured. For instance, if purchases and promotional emails are tracked at a daily level, it can be difficult to discern if a customer who received an email on a particular day and also maede a purchase the same day received the email before she made the purchase

Question 21

Simple random sample

Answer 21

takes a random draw from the target population using for instance a coin flip or a dice roll. This gives to each subject an identical probability of entering the sample, ensuring that the sample will be representative of the target population

Question 22

cluster sample

Answer 22

when it is easy to measure groups or clusters of subjects, randomly sample from among the clusters

Question 23

stratified sample

Answer 23

use a procedure to make sure that the sample contains different types of subjectscon

Question 24

convenience sample

Answer 24

sample in some way that is easy for the researcher, e.g. an academic might conduct the experiment with students or a company might conduct the experiment using store locations that are nearby

Question 25

interaction effect

Answer 25

when the combined effect of two treatments is better than the sum of the individual effects, there is an interaction effect. Detecting interactions is the main reason why companies use multivariate tests. In addition multivariate tests can reduce required sample sizes and increase the amount that can be learned in the time frame of a single test.

Question 26

An experiment requires two stages

Answer 26

a design state before the experiment is implemented, including, for instance, the selection of the initiative as well as the number and randomization of groups, and an examination stage after the experiment took place to analyze the data which had been generated

Question 27

The term A/B testing is often used when

Answer 27

referring to field experiments in online environments meaning that for example, one randomly selected group of customers is presented with packaging design A and another gorup is presented with design B for an otherwise identical product

Question 28

Control over treatment

Answer 28

describes the researchers ability to design and have full control over the type of treatment and the selection of subjects and circumstances

Question 29

Randomness of assignment

Answer 29

describes whether the experimental design classifies subjects randomly (or not) to the treatment and the control group. Here we differentiate between random and nonrandom assignment

Question 30

The differences in differences methodology

Answer 30

exploits two different comparisons within one design: An intertemporal comparison and a cross sectional comparison. The first comparison is the intertemporal difference within the treatment group; thus, the difference in observation before and after the treatment only for the gorup which has been affected by the manipulation

The second difference constitutes the variation between the treatment and the control group. here, observable outcomes of treatment and control group are compared for one or more points in time only after the intervention, that is, a cross sectional difference is exploited. In combining both differences, intertemporal and cross sectional, the differences in differences methodology allows to draw causal inferences as long as it fulfills specific conditions

Question 31

Staggered designs

Answer 31

here researchers either design their field experiment in a way, or exploit the fact that treatment does not occur to all treated individuals or groups at the same time but is introduced step by step. Staggered designs are often observed when regulations or legal directives are adopted or firms subsequently roll out a new policy over subjects such as business units, facilities, employee groups or customersT

Question 32

Three more assumptions with relevance to the differences in differences method and provided reasoning to which extent the sepcific research design meets those assumptions

Answer 32

The treatment status of one unit should not interfere with another units outcome. FOr example customer As status (qualified for free shipping) should not affect the amount of money spent by another customer B.
Neither subject in the treatment or in the control group anticipates the intervention or is affected by the treatment prior to the intervention. This assumption might be violated in cases where subjects can voluntarily choose to become part of a group, that is, a customer may decide to purchase an additional item to cross the threshold of the free shipping minimum order amount
Finally, the difference in differences method implicitly assumes perfect compliance. This means all individuals in the treatment group are in fact treated. In case of a violation, the method estimates (just) an intention of treat effect. THis intention of treat effect is more conservative than the treatment on the treated estimation because it is based on a smaller variance within the data set making it harder to detect a statistically significant effect between treatment and control group. For free shipping threshold it may occur that certain customers may either simply not recognize that free shipping is available or at which threshold the would qualify for the free shipping