4. Testing at High Tempo Flashcards
What year did Art Briles become the head coach of Baylor University football team?
2007
What was the average points scored by the Baylor Bears under Art Briles?
64 points a game
What type of offense did Baylor implement that contributed to their success?
High-tempo, no-huddle offense
How many more offensive plays did Baylor run per game compared to their competitors?
13 more offensive plays
What is the percentage increase in the number of plays run by Baylor compared to the average college team?
20 percent increase
What is the primary goal of the high-tempo growth hacking process?
To learn faster
True or False: Most experiments in growth hacking produce significant results.
False
What is the relationship between small wins and overall success in growth hacking?
Small wins compound over time to create big successes
What happens to profits with a 5 percent increase in customer retention?
Increase in profits of between 25 and 95 percent
What is the recommended number of experiments leading growth teams should run weekly?
20 to 30 experiments
Fill in the blank: The stages of the growth hacking process include data analysis, ________, experiment prioritization, running experiments, and analyzing results.
idea generation
What is the North Star metric for the hypothetical grocery app team?
Monthly revenue per shopper
What is the first step the growth team leader should take when forming a new team?
Pull staff from marketing, engineering, product, and data science groups
What feature of the grocery app is considered the ‘aha moment’ for users?
Convenience of ordering groceries on your phone
What is the growth equation determined by the grocery app team?
NUMBER OF INSTALLS × NUMBER OF MONTHLY ACTIVE USERS × NUMBER OF PURCHASERS × AVERAGE ORDER SIZE × REPEAT PURCHASE RATE
What is the average order size for avid shoppers on the grocery app?
More than $50
What type of analysis does the growth lead conduct in the Analyze stage?
Dive into the data to identify distinctive groups
What type of feedback is crucial for the growth team during the Ideate stage?
User surveys and interviews
True or False: Self-censorship is encouraged during the ideation process.
False
What should the growth lead establish to manage ideas and track results?
A project management system
What characteristic differentiates regular shoppers from those who rarely use the grocery app?
Regular shoppers have distinctive behaviors and characteristics
What is the goal for the volume and tempo of experiments set by the growth team?
To determine how many tests they can reasonably manage
What is the role of a user experience designer in a growth team?
Propose changes to the design of certain screen displays
What should the growth lead set up to coordinate idea management?
A project management system
What is a key tenet of growth hacking?
Cross-functional collaboration and sharing of information
What software program did GrowthHackers create for idea management?
Projects
What should ideas be submitted to?
An idea pipeline
Why is it important to standardize the format of idea submissions?
To evaluate ideas quickly without needing lots of questions
What should submissions articulate clearly?
The change to be tested, reasoning, and how results will be measured
What is an example of an idea aimed at enticing new users?
Encouraging users who downloaded the app but haven’t purchased yet
What is the proper format for submitting an idea?
IDEA NAME, IDEA DESCRIPTION, HYPOTHESIS, METRICS TO BE MEASURED
What should the IDEA NAME be limited to?
Under 50 characters
What should the IDEA DESCRIPTION address?
Who, what, where, when, why, and how of the idea
What does the HYPOTHESIS in an idea submission represent?
A simple proposition of expected cause and effect
What is an example of a hypothesis for the shopping list feature?
“By making it easier for shoppers to reorder items, repeat purchases will increase by 20 percent.”
What metrics should be measured to evaluate the outcome of a test?
Multiple metrics including usage of the feature and purchasing behavior
What is the goal of generating a large volume of ideas?
To increase chances of finding winning ideas that spur growth
Who should be encouraged to contribute ideas?
Team members, sales team, marketers, outside suppliers, partners, and customers
What is the ICE score system used for?
To rank ideas based on Impact, Confidence, and Ease
What does the ‘I’ in ICE score stand for?
Impact
What does the ‘C’ in ICE score stand for?
Confidence
What does the ‘E’ in ICE score stand for?
Ease
How is the ICE score calculated?
By rating each idea on a ten-point scale for impact, confidence, and ease
What should the team prioritize when selecting ideas to test?
Ideas with higher ICE scores
What is the importance of confidence in the ICE scoring?
It measures the submitter’s belief in the idea’s effectiveness based on evidence
What is a challenge of scoring one’s own ideas?
It requires relative subjectivity and predicting future outcomes
What should ease measure in the context of an experiment?
The time and resources needed to run the experiment
What is a potential downside of overly ambitious ideas?
They may require too much time to implement
What is the role of the growth lead in the scoring process?
To review initial scores and suggest modifications based on experience
True or False: The aggregate score of an idea is always the ultimate determinant of which ideas are selected for testing.
False
What is the purpose of the scoring system in growth experiments?
To use it for relative prioritization of ideas and experiments
The scoring system helps teams focus on the most promising experiments without getting bogged down in perfecting scores.
What should the growth lead do when there is uncertainty about a score?
Use her best judgment and act decisively
This helps keep the team moving forward without getting stuck.
What is the significance of the 700 percent jump in sign-ups from the sign-up form experiment?
It illustrates that low-rated experiments can yield unexpected high results
This emphasizes the unpredictability of experiment outcomes.
What does the ICE scoring system stand for?
Impact, Confidence, Ease
This system helps in evaluating ideas quantitatively.
Name one alternative scoring system to ICE.
TIR (Time, Impact, Resources)
Another example is PIE (Potential, Importance, Ease).
What should be done with ideas that cannot be launched during the current week?
Store them in the experiment pipeline
This allows for future consideration and prioritization.
What is the role of the marketing team member in the first-time-shopper promotion experiment?
Manage the promotion
This includes coordinating with other teams for implementation.
How many test candidates should each team member nominate for the growth meeting?
A maximum of three test candidates
This keeps the discussion manageable.
What happens to experiments after they are selected for testing?
They are moved to the Up Next queue
This indicates they are ready for preparation and deployment.
What is a critical guideline for ensuring reliable test results?
Design every experiment to produce statistically valid results
This is essential for accurate learning and decision-making.
What statistical confidence level is recommended for testing?
99 percent
This significantly reduces the risk of false-positive results.
What should be done when results of an experiment are inconclusive?
Stick with the original, or control, version
This minimizes the risk of implementing a potentially poor variant.
What key elements should a test summary include?
Name/description of the test, type of test, features impacted, key metrics, test timing, hypothesis and results, confounding issues, conclusions
This comprehensive summary aids in future analysis and learning.
What is the purpose of a knowledge base in growth testing?
To store and easily search results of all experiments
This prevents repetition and aids in revisiting previous tests.
What type of communication helps keep employees informed about growth process updates?
Regular updates via email or instant messaging channels
This fosters transparency and engagement within the company.
How often should growth meetings be held?
Once a week
Some teams may opt for bi-weekly meetings depending on their schedule.
What should the growth lead review prior to the growth meeting?
The number of experiments launched, key metrics, data about concluded tests, high-level assessment of previous activities
This ensures informed decision-making during the meeting.
What should the agenda for the growth meeting include?
A compilation of relevant information and a living document shared with the team
This facilitates structured discussions and follow-ups.
What is the purpose of gathering data about tests that were concluded?
To conduct a high-level assessment of the previous week’s activity and results
This includes assessing both positive and negative effects on growth discovered from the experiments.
What is a living document shared with the team before meetings?
The meeting agenda
Some teams keep this document in cloud storage like Google Docs or Dropbox, while others use internal wiki pages.
What is the focus of the first 15 minutes of the meeting?
Metrics review and update focus area
What is the North Star metric?
The key growth metric reviewed during the meeting
It helps determine whether things are going well or not.
What are key positive factors in the metrics review?
Improvements in metrics resulting from experiments or external factors
Examples include improved reorder rates or influx of new app users.
What are key negative factors in the metrics review?
Drop-offs in performance and issues holding back growth
What should the growth team focus on during the meeting?
The user experience or growth lever they are focused on
This may involve confirming or discussing shifts in short-term objectives.
What is the purpose of the 10-minute review of last week’s testing activity?
To assess the tempo and discuss tests that were not launched
What does ‘tempo’ refer to in the context of testing activity?
The number of experiments launched compared with the team’s tempo goal
What is discussed regarding tests that were not launched?
Reasons for delays in launching those tests
What is covered in the 15-minute segment on key lessons learned?
Preliminary and conclusive results from tests launched
This includes team assessment of experiments and implications for further action.
What happens during the selection of growth tests for the current cycle?
Team discusses and nominates experiments to launch
The growth lead assigns owners based on expertise.
What does the growth lead report on after discussing tests?
The number of ideas waiting for consideration or slated for future launch
What is a significant advantage of growth hacking?
Speed of producing significant improvements
What can happen within two weeks in the growth hacking process?
A hack can go from germ of an idea to growth driver
What is an example of a successful experiment mentioned?
A $10 special offer for making a first purchase increased conversion by 15 percent
What might the team discover from analyzing results?
Insights leading to further experiments to increase purchase value
What do failed experiments provide in the growth hacking process?
Significant learning opportunities
What might the team test if a free delivery offer at $50 fails?
A lower threshold for free delivery, such as $40 or $35
What will Part II of ‘Hacking Growth’ focus on?
Major levers for growth: user acquisition, activation, retention, and monetization
