Lecture 10 Impact and effectiveness of interventions

Question 1

Scientific
Experiments

Answer 1

Experiments should test the effectiveness of a product and how it is
received. This is usually done after the product is built and deployed with real users.

Question 2

Vetting ideas –
first steps

Answer 2

Quick studies with competing prototypes or product concepts can vet ideas at low cost.
Do this before the product is built.
Vetting your ideas early on can prevent you from investing resources and effort into something that will not work.

Question 3

Optimizing

Answer 3

When you are working on a complex product, test different parts individually during development to refine and to understand the impact of each feature individually

Question 4

Drift and long-term effects

Answer 4

Some interventions work simply because they are new—like reminders or novel features.
Some products might not seem effective short-term but have longterm benefits.
You can re-run an initial experiment to check if it has lost its effectiveness.

Question 5

Regression toward the mean

Answer 5

If the result of one test is extreme, it is more likely that the next result will be less extreme. When random factors (luck, context) are influencing your test results, then you will likely see a result closer to the average on the next test.

Question 6

Randomisation

Answer 6

We want to make sure that the groups are comparable in other random characteristics (e.g., personality, demographics, preferences).
We are likely to achieve this through randomization.

Question 7

Randomized Control Trial

Answer 7

• Most effective and rigorous tool for determining the impact of your
  intervention on behaviour, experience, health outcomes etc.
• If the sample and test are not biased, it provides a reliable result.

Question 8

Steps to follow when evaluating your persuasive technologies

Answer 8

1. Outline what the goal of the intervention is.
2. Find the best currently existing intervention for reaching this goal (gold standard).
3. Relying on prior work, clarify how you can measure the outcome (operationalization).
4. Find test subjects and randomly assign them to your groups: a) your intervention group compared to b) the gold standard and/or c) a control group without an intervention.
5. Deploy the intervention(s) depending on the groups.
6. Measure the outcome and perform an appropriate statistical analysis.
7. Interpret the results based on substantial theoretical knowledge.

Question 9

Outline the goal of the
intervention

Answer 9

1, The outcome
2, The intervention
3, The target audience

Question 10

Your comparison(s)

Answer 10

1, Gold standard - your intervention might work, but is it better than the best intervention that is readily available?
2, Control group - symptoms might simply improve over time (e.g., regression to mean); being part of a study can lead to placebo effects.

Question 11

Operationalization

Answer 11

How can you best measure and quantify your desired (and undesired) outcome?

Question 12

Randomly assign the audience

Answer 12

Sample enough people from the target audience. Randomly assign them to at least two same sized groups. We will call them a control group and a treatment group.

Question 13

Measure the outcome

Answer 13

After enough time has passed, measure the outcomes for each group to compare the effectiveness and direction of change (improvement, detrimental effects, etc.)

Question 14

Interpret the results

Answer 14

If the impact is large enough, you can conclude that your intervention is practically and statistically meaningful.

Question 15

Cross-sectional study

Answer 15

Data collected from many people once

Question 16

Longitudinal study

Answer 16

Data collected from people multiple times over longer intervals (e.g. months, years)

Question 17

Baseline

Answer 17

The average outcome for people who don’t have the product (feature, technology, or intervention).

Question 18

Noise

Answer 18

The variance in outcomes (how far do data points spread from the average) among people without the product.

Question 19

Effect size

Answer 19

How much of an effect do you reasonably expect from the intervention on the outcome to call it a success?

Question 20

Alpha (α) Error (Type I Error)

Answer 20

The probability that we reject the null hypothesis even
though it is true.

Question 21

Beta (β) Error (Type II Error)

Answer 21

The probability that we accept the null hypothesis
even though it is false.

Question 22

(Measure the baseline &) Deploy the intervention.

Answer 22

Offer your intervention to the treatment group.
* Control: Do not give the intervention to the control group or give them something neutral that should not work.
* Gold Standard: Offer the best alternative intervention to another group for comparison.

Question 23

Simultaneous impact

Answer 23

Also known as A/Null tests.
A randomly selected group receives a new feature or product, and the other group does not receive it.
Tests for the impact of a new feature or product.

Question 24

Simultaneous comparison

Answer 24

Also known as an A/B test.
Randomly selected groups receive two different versions of a product
The result shows the difference in outcomes between those two versions.
It is a simple randomized control experiment.

Question 25

Multiarm comparison

Answer 25

A simple extension of an A/B test that looks at more than two versions at the same time.
Each version gets its own randomly assigned and selected group of
participants, all from the same pool of people.
This is called an A/B/C (/D etc.) test, for obvious reasons.

Question 26

Staggered rollout

Answer 26

In a staggered rollout, each group eventually receives the intervention.
However, one randomly selected group receives the intervention earlier than the other.
The only difference between those two groups should be the prior exposure to the intervention.

Question 27

Non-product-related causes for changes

Answer 27

• Time
• Experience
• Data availability or quality
• Composition of the population