Product Execution & Analytics

Question 1

What part of the PM role frequently involves analytics?

Answer 1

Assessing the impact of a feature

Question 2

Why is it important to predict the impact of features ahead of time (before launch)?

Answer 2

So you can make feature prioritization decisions.

Question 3

How do analytics tend to change as a company grows/matures?

Answer 3

More mature companies tend to have more and higher-quality data available about all aspects of itself, so everything becomes more and more quantified.

Question 4

What are the four common contexts in which analytics appears in a PM interview?

Answer 4

• Assessing a product
• Optimizing a user flow via funnels
• Running feature experiments via A/B testing
• Investigating a metrics anomaly

Question 5

What does the TOFU acronym represent in product analytics?

Answer 5

The four major buckets of metrics for monitoring product health.

Question 6

List the four major buckets of product health metrics (TOFU)

Answer 6

• Tech (technical infrastructure)
• Objects (key objects in the product)
• Finance (financials/business metrics)
• Users (user behavior, engagement)

Question 7

Give some examples of technical infrastructure metrics

Answer 7

• Page load times
• API calls
• Bandwidth usage

Question 8

Explain what key objects metrics are and give an example.

Answer 8

Generally every product has some key objects that it is centered around; something the user transacts or manipulates—some pieces of content that may or may not be user generated. These metrics are often used to optimize the content of the product (rather than its structure).

For example, a streaming app like spotify might track how many songs there are, what proportion of the overall library is played, or which songs are most popular.

Question 9

Give some examples of financial/business metrics.

Answer 9

Revenue, customer acquisition cost (CAC), churn rate, lifetime value.

Question 10

What’s the difference between annual recurring revenue and annual run rate?

Answer 10

Annual recurring revenue represents actual recurring revenue commitments from customers, while annual run rate represents an annualized projected revenue that may not be sourced from subscription revenue.

Question 11

What is the HEART framework for the categories of user behavior metrics?

Answer 11

• Happiness
• Engagement
• Adoption
• Retention
• Task Success

Question 12

Give an example of a user happiness metric.

Answer 12

Net promoter score (“On a scale of 1-10, how likely are you to recommend this product to a friend or colleague.”)

Question 13

What’s the difference between an adoption metric and an engagement metric?

Answer 13

Adoption metrics tend to be near the top of the funnel (number of sign-ups) while engagement metrics tend to be more mid-funnel (e.g. daily active users).

Question 14

How is conversion rate defined?

Answer 14

It’s effectively the transmission coefficient between two stages of a funnel, but it is up to product managers to define which two stages most meaningfully can be labeled the “conversion rate”.

Question 15

Explain the “task success” part of the HEART framework of user behavior metrics.

Answer 15

This subcategory of metrics captures how many users “succeed” at receiving the core value of the product. It is up to the product manager to decide what success means for any given product. For AirBnB it may be the percentage of users that book a stay.

Question 16

What makes an experiment successful?

Answer 16

It was well defined, correctly executed and provided an accurate, statistically significant insight.

Question 17

What are the benefits of explicitly stating a hypothesis before performing an experiment?

Answer 17

It forces clarity, gives team members a chance to make a prediction or “put a stake in the ground”, and helps ensure that you set up the treatments in a way that tests the hypothesis.

Question 18

When choosing treatments for an experiment, what two factors should limit how many treatments you select?

Answer 18

How many treatments you actually need, and whether you have enough users to properly evaluate all those treatments.

Question 19

What is a perverse incentive in product experimentation that you should look out for?

Answer 19

The incentive to make experiments appear successful (which may cause you misinterpret the results or change which experiments you choose to do).

Question 20

What is a good MECE framework for beginning to investigate a metric anomaly?

Answer 20

Dividing potential causes into internal factors and external factors.

Question 21

Give some examples of internal factors that could explain a metrics anomaly.

Answer 21

• Data integrity (e.g., is the metric correctly reported?)
• New release (e.g., did we release code that broke something?)
• Experiments (e.g., is an experiment having negative externalities?)
• Infrastructure failure (e.g., are our own systems going down?)

Question 22

Give some examples of external factors that could explain a metrics anomaly.

Answer 22

• User driven events (e.g., is there a flood of user driven activity on our system due to some event?)
• Holidays / major cultural events (e.g., is it July 4th or the SuperBowl? )
• Third party infrastructure (e.g., is there an AWS outage?)
• Security breaches (e.g., are we under DDoS or some other type of attack?)

Question 23

When investigating a metrics anomaly, what sort of probing questions could you ask to find out more about which users are impacted?

Answer 23

• Is it affecting all users or just some sub set of users?
• Is the effect isolated to certain platforms (e.g., only on iOS or Android or web)?
• Is it dependent on the browser the user users?
• Is it dependent on the version of the app they’re on?

Question 24

When investigating a metrics anomaly, what sort of probing questions could you ask to find out more about the nature of the change?

Answer 24

• Did the change occur gradually or suddenly?
• Have we seen a change like this in the past? If so, in what context, or what was the cause?

Question 25

When investigating a metrics anomaly, what are some probing questions oen could ask to learn more about how the change was discovered?

Answer 25

• Did a customer submit a bug report or service ticket? What else does the customer know?
• Was the change noticed in our dashboards?
• Is the metric derived from other more simple metrics that we can decompose it into?

Question 26

List examples of changes relating to a product that could explain a metrics anomaly.

Answer 26

• New code
• Experiments
• Updates to infrastructure
• New customer support policies
• Changes in competitors products
• New press coverage or other publicity

Question 27

List four common applications of funnels.

Answer 27

• Sign up
• Conversion
• Sharing
• Purchase

Question 28

What does LTV stand for and what does it mean?

Answer 28

Lifetime value. It defers to the sum of discounted cash flows that the company expects to receive from the average customer.

Question 29

Does the LTV formula account for customer acquisition cost (CAC)?

Answer 29

The way that Rocketblocks defines it, yes. But in general just make sure you are clear about how you intend to use it.

Question 30

What is a north star metric?

Answer 30

A high level metric that indicates the health of a product or component of a product that a particular team is responsible for. E.g. nights booked for AirBnB.

Question 31

When asked “what would you measure”, you should make sure that the metric you choose is…

Answer 31

• Specific
• Relevant
• High Priority / Focused

Question 32

What is an “aha moment” metric, and why is it useful?

Answer 32

A user behavior that is a highly specific (and, ideally, sensitive) indicator that the user will be retained. It is especially useful because it’s a leading indicator of retention, which normally takes lots of time to measure.