How to Design for Experimentation Flashcards
Why are designers frightened by sharing unfinished or imperfect features?
Because we’re used to spending time on small details or thinking about products more holistically.
How is shipping unfinished or imperfect things in agile usually misinterpreted?
“We routinely ship bad features that aren’t well thought out because we only care about velocity.”
What are Beta Testers?
Inviting a small group of real users to try out a new feature or product can get you real input about what works and what doesn’t without the danger of releasing it to everybody in your user base.
What is the benefit of using Beta Testers?
But when you allow small groups of people to opt into new features with the express purpose of getting feedback on changes, you get the best of both worlds—information about real customer experience and minimal disruption for people who don’t have time to test your new, possibly unfinished features.
What are Partial Rollouts?
Even after you’ve tested with your opt-in group, it’s still often a good idea to test with a small group of regular users. You can do this by rolling out the feature to a small percentage of the user base, maybe no more than 10%, or sometimes even less depending on the absolute numbers of people in your user base. This can help you look for any problems that weren’t caught in beta and to uncover likely significant customer complaints that you weren’t expecting.
After you’ve tested with a partial rollout and hopefully iterated to improve the feature, you may roll out to everybody, or you may just keep including people gradually until everybody has access to the new feature.
What are Feature flags?
A common way for teams to control how many people see a particular feature and when is to use something called “feature flags.” Feature flags are a technical tool that engineers can add to features that allow them to be turned on or off almost instantly. Many teams have a control panel for new features that product managers can use to manage the users who can currently see a particular feature.
How Does This Effect Designers?
Remember, really agile teams are cross-functional and collaborative. That means that we should all have some input into things like whether a feature is ready to be released to all of our users. Often, designers and researchers can take advantage of partial rollouts and beta test groups to get important user feedback on our designs. We can also strongly recommend that something get feedback from smaller groups of people before general availability (sometimes called GA), because we’re concerned that there may be some issues we’ve missed.
It’s also important for us to remember to design and release features in a way that allows for partial feedback. For example, if there is a particularly risky design or something that we feel we need to see people use in real life, we can start our designs with that and test it early. If it’s wrong, we can iterate and change it quickly before we get too far into the feature.
There are lots of ways to get working features in front of real users without constantly changing the interface for everybody. What are some of these methods?
Using beta testers, doing partial rollouts, and using feature flags
What can you NOT expect to learn from beta testers?
a. If there are significant bugs and usability issues.
b. If a feature will be successful with a majority of users.
c. If a feature is usable and understandable for target users.
b. If a feature will be successful with a majority of users.
When your team releases updates to beta testers, it is the team’s responsibility to ensure that unsafe features are not released. Users who opt into beta test programs should be protected from any security flaws or potential data loss. Beta testers are usually forgiving of small bugs issues, but if there is a significant problem, you should find out and fix it quickly. What beta testers cannot help you with is predicting whether a majority of users (who didn’t opt for the beta program) will be happy with the features.
If you wanted to find out whether people will actually use a feature, how can you do that in an agile team?
a. Create high-fidelity prototypes and recruit random people to test it out.
b. Develop the feature and conduct guerilla testing with random people on the street.
c. Build a small version of the feature and feature flag it in order to roll it out to a small set of users.
c. Build a small version of the feature and feature flag it in order to roll it out to a small set of users.
Creating high-fidelity prototypes and guerilla testing are great for usability tests. But they will not be able to reveal if users will actually adopt a risky idea. The advantage of agile teams is that you can selectively roll out the feature to real users and track how they use them in their context. This will also remove any biases that may crop up when you test in-person, as people will not feel like they are being judged or feel the need to give positive feedback.
If the idea is successful in early releases, you can build on it, and if not, you can quickly switch it off.
If you wanted to find out whether people will actually use a feature, how can you do that in an agile team?
a. Create high-fidelity prototypes and recruit random people to test it out.
b. Develop the feature and conduct guerilla testing with random people on the street.
c. Build a small version of the feature and feature flag it in order to roll it out to a small set of users.
c. Build a small version of the feature and feature flag it in order to roll it out to a small set of users.
Creating high-fidelity prototypes and guerilla testing are great for usability tests. But they will not be able to reveal if users will actually adopt a risky idea. The advantage of agile teams is that you can selectively roll out the feature to real users and track how they use them in their context. This will also remove any biases that may crop up when you test in-person, as people will not feel like they are being judged or feel the need to give positive feedback.
If the idea is successful in early releases, you can build on it, and if not, you can quickly switch it off.
How do you create a good MVP?
You have to deeply understand your user’s problem and then identify the core of a solution.
What is the only way to build an MVP that will help you learn whether your solution is the correct one for any given user problem?
Finding the core of a feature or product.
It helps you optimize for selecting both the “minimum” and the “viable” parts of the MVP.
You have to understand that if a product doesn’t solve a problem for users and help the team learn something, it’s not really _______. And if it’s full of extra features that are just nice to have, it’s definitely not _______!
viable, minimum
What is the goal of an MVP?
Our goal on agile teams is to avoid waste when possible, and one way we do that is by building small useful things that we can then build on and make better and more useful over time. Sometimes, though, we avoid waste by building small things and then throwing them away when it turns out that they’re not the right direction or we find a much better solution. The goal of the MVP is to help us decide which of these paths to take.
