Web Analytics & Testing

Question 1

Web Analytics Five Key Dimensions

Answer 1

Clickstream, 
Outcomes,
Experimentation, 
Customer Voice (UGC, Social listening and online ratings)
Competitors

Question 2

Paywall

Answer 2

Subscribers get something different than free customers

Question 3

Freemium

Answer 3

low quality or limited amount of product for free.

Question 4

Clickstream

Answer 4

Just the first level of info, need to amplify insights. 
What pages did people visit?
What products did people purchase?
What was the average time spent?
What sources did they come from?

Question 5

Web Analytics 2.0

Answer 5

Clickstream
What pages did people visit? Etc…
Outcomes
How much revenue was generated?
How much were costs reduced?
How much loyalty do users show?
Experimentation
Why do user behave as they do?
What drives behavior?
Voice of the Customer
Ask the customer.
Surveys, Usability Testing, UGC.
The Competition
Competitive Intelligence, Industry
Benchmarks

Question 6

Outcomes

Answer 6

How much revenue was generated?
How much were costs reduced?
How much loyalty do users show?

Question 7

Experimentation

Answer 7

Why do user behave as they do?

What drives behavior?

Question 8

Voice of the Customer

Answer 8

Ask the customer.

Surveys, Usability Testing, UGC.

Question 9

Time on site

Answer 9

Be careful, you may want low time for customer satisfaction!

Question 10

The Competition

Answer 10

Competitive Intelligence, Industry

Benchmarks

Question 11

Click Stream Foundational Metrics

Answer 11

Visitors / Unique visitors
§ Pay attention on definition of
“unique” (cookie? date? IP?)
§ Time on site
§ Tricky! Should consider the goal of
the site
§ Page views
§ Good for content/brand sites
§ Unclear for other sites
§ Increasingly outdated
(blogs, Gmail, Flash, dynamic content)
§ Session data
§ PV/Session
§ Bounce rate (“One-and-Dones”)
§ Reveals real visitors
§ % of single page visits
§ (or % of <5 second visits) Segment, segment, segment!

Question 12

GA

Answer 12

Google Analytics

Question 13

Segmenting Clickstream

Answer 13

Don’t look at the average user, look at the important segments for basic analytics.

Question 14

Goals for Clickstream!

Answer 14

“Unique Visitors” tends to be THE
metric to follow, BUT instead:
§ Set up Goals and measure
Conversion Rate and Goal Value
(SettingsàEditàGoal)
§ Segment by:
§ Referring sites
§ Search engines + Keywords
§ AdWords campaigns
• Analyze for Leads!
– “Wikipedia referrals are more engaged and have low bounce rate”

Question 15

Clickstream content analytics

Answer 15

Top Content
§ Why users are coming
§ What they are looking for
§ Top Landing (Entry) Pages
§ First impression!
§ Polish and direct users to goals
§ Click Density Analysis
§ E.g. CrazyEgg.com
§ Funnel Analysis
§ In multi-page processes, where do users abandon?
§ Mortgage application at Agency.com à move personal information form later (after complex work of mortgage with easier stuff)
§ Abandoned carts / purchases at Lane Bryant à offer free shipping; did some surveys and found sticker shock on shipping price.

Question 16

You can’t manage…

Answer 16

You can’t manage what you don’t measure

Question 17

o Clickstream referral metrics

Answer 17

Where are users coming from?
§ How does traffic from different referrers behave?
§ How well is this measured?
§ What about Social Referrals?

Question 18

Bot Traffic

Answer 18

 >60% non-human traffic
 Search Engine (Good) Bots (30%): Indexing the Web
 Bad BOTS: impersonators, spammers, hacking, scrapers
 You must NOT include the BOTs in your analytics
 - Otherwise you’ll tailor your analysis to BOTs and optimize to increase them
 - BOT detection algorithms are needed; these tend to be pretty accurate (~80-90%)
 -Off the shelf algorithms you can download from universities or DS can build one
- Also Google Analytics can remove BOTs (fairly accurate)

Question 19

Dark Social in Social Referrals

Answer 19

We know it’s coming from social, but referral source not being passed to the website (70+%)

Question 20

5 Critical App Metrics @ Humin

Answer 20

 § Growth § DAU/MAU § # Users § # Relationships § Retention § A7, A15, A30, A45 (how many new starters are still with us X days later) § Measured in Daily Cohorts § Engagement – Key User Actions / DAU § Profile Opens § Call/Text §

Question 21

DAU

Answer 21

Daily active users

Question 22

Humin Aha! moments

Answer 22

Profile Opens, Call/Text swipes, Voicemail and Location Services (Enablers)

Question 23

FUE Funnel

Answer 23

First user experience funnel

Question 24

Importance of Randomization

Answer 24

90% of randomized can be replicated, only 20% non-randomized. Jim Manzi, Uncontrolled

Question 25

A/B vs. multivariate testing

Answer 25

A/B is one: e.g. Zynga optimizing customer acquisition funnel
Multivariate Testing: Randomize elements and find impact of all together and interactions of different modules. So best combinations.

Question 26

The cost of knowledge

Answer 26

A/B Testing always exposes users to a lower performing version for some period of time

Question 27

Segments and testing

Answer 27

Post-hoc analysis

Question 28

Stratified Random Sampling

Answer 28

Population weighted random sampling or over-weighting

Question 29

Best Test Practice OEC

Answer 29

Establish the Overall Evaluation Criterion (OEC)
– Agree early on what you are optimizing
– Getting agreement on the OEC in the org is a huge step
forward
– Suggestion: optimize for customer lifetime value, not
immediate short-term revenue/growth
– Criterion could be weighted sum of factors, such as
• Time on site (per time period, say week or month)
• Visit frequency
– Report many other metrics for diagnostics, i.e., to
understand why the OEC changed and raise new hypotheses

Question 30

Best Test Practice

Answer 30

Run A/A tests – simple, but highly effective
– Make sure they’re truly different for the OEC @ 95% significance: validates operations, execution, samples, etc. EVERY TIME you run the experiment
- Run an experiment where the Treatment and Control
variants are coded identically and validate the following:
1. Are users split according to the planned percentages?
2. Is the data collected matching the system of record?
3. Are the results showing non-significant results 95% of the
time?
This is a powerful technique for finding problems
– Generating some numbers is easy
– Getting correct numbers you trust is much harder!

Question 31

Best Test Practice: Ramp Up

Answer 31

Ramp-up
§ Start an experiment at 0.1%
§ Do some simple analyses to make sure no egregious
problems can be detected
§ Ramp-up to a larger percentage, and repeat until 50%
§ Minimum sample size is “quadratic in the
effect” we want to detect
§ Detecting 10% difference requires a small sample and
serious problems can be detected during ramp-up
§ Detecting 0.1% requires a population 100^2 = 10,000
times bigger
§ Abort the experiment if treatment is
significantly worse on key metrics

Question 32

o Min sample size is quadratic in the effect we want to detect

Answer 32

 Detecting 10% difference requires a small sample
 Detecting 0.1% requires a population 100^2 = 10,000 bigger vs. 10% difference.
 Abort the experiment if treatment is significantly worse on key metrics

Question 33

How to see your travel on the internet

Answer 33

Google Maps Timeline