A Beginner's Guide to Product Experiments
By Catherine Huang and Joyce Chen
This article is a free preview of the Passport Resources library. To access our entire library of startup best practices, click the button above to sign up for your free Passport account.
Product development is all about working to create the best product possible. In order to build and maintain a uniquely valuable product, you’ll need to run effective experiments. Throughout this process, you’ll discover important learnings that will allow you to iterate more quickly and effectively.
A product experiment is essentially applying the scientific method to product development, starting with a hypothesis and running through a series of tests to arrive at insights. Data-driven insights empower you to build a uniquely valuable and impactful product.
To help us learn more about how to run product experiments, we spoke to Catherine Huang, a product leader and founder of Folio.works.
In the video below, Catherine walks us through the basics of product experimentation.
As Catherine explained, a product experiment is the process of testing a hypothesis about your product. Many experiments can and should be conducted before you start to build any aspect of your product. Catherine suggests running a series of minimum viable tests (MVT) on your riskiest assumptions that will either lead your company to success or failure. Your minimum viable product (MVP) will become a part of these tests as well. By conducting these experiments, you’ll collect valuable insights and save significant money on development.
Designing the Experiment
To begin preparation for your experiment, start with a clear product vision. Identify the problem that you’re looking to solve, and the impact you hope to create. From there, you’ll be able to define the measures of success, or key performance indicators (KPI). Then, create the test itself. Make sure that the test you’re creating is valid by checking for confounding variables. In the beginning, you should aim to create the smallest, most focused test possible. Lastly, you’ll want to check your results. Whether or not you’ve proven your hypothesis, you’ll have learned something valuable. Interview users to understand the “why” behind their decisions or responses, then create a new hypothesis to continue testing.
Some questions you might ask to form your hypotheses include:
-
Demand
-
Does this product provide significant value?
-
Does this product solve a painful issue?
-
Are customers willing to pay money to relieve that issue?
-
-
Pricing
-
How much are customers willing to pay for this product?
-
-
UI/UX
-
Is this the right place to insert this product in the user flow?
-
Is this the right experience?
-
Is this the right medium?
-
Where and when do users engage with this product? (ie. during their morning commute, after dinner, etc.)
-
What life events prompt users to engage with your product?
-
Prioritizing Experiments
For many early-stage founders, resources are limited. So, you really want to focus on running experiments that will help you the most. Consider this matrix with effort on the x-axis and impact on the y-axis:
.png)
You want to target the experiments that will give you the highest amount of impact for the least amount of effort.
For instance, a painted door test on pricing could look like creating a quick no-code landing page with a description of your product with its price, and a button to purchase. You’ll be able to track the amount of people who are willing to pay that price for your product, even before you spend lots of resources building it. In other words, with just a little bit of effort, you’ll be able to test a critical assumption that makes a big impact in the success of your product.
On the other hand, an A/B test on brand messaging could require the work of a graphic designer, and money to be spent on advertising. But, the results of this test do not impact the product development process very much. With limited resources, this should be a low priority test, since it requires a large amount of effort for very little impact on the success of your product.
Common Mistakes
As you begin running your own product experiments, steer clear of some of these common mistakes that founders make.
-
Confounding Variables
Confounding variables are additional factors that could affect the outcome of your test, leading to inaccurate results. For instance, in an A/B test for messaging, you may use two different product descriptions to see which one performs better. However, you test it on two different platforms, Instagram and Twitter. Your results may show that the description on Instagram performed better. While you may be tempted to make the assumption that it was the description that caused the better performance, the confounding variable of different platforms makes this inaccurate. To avoid this, limit the amount of differences between your tests and focus on changing one aspect at a time.
-
Too Many Tests at Once
Just like with the risks of confounding variables, you don’t want to run too many tests at once. When you’re developing quickly, it may be tempting to combine as many tests as possible to make the process more efficient. However, you’ll often end up with inaccurate results because you won’t actually be able to attribute any of the results to a specific cause. It can also be difficult to keep track of all the variables. Keep tests as simple as possible, and focus on running the tests that will make the biggest impact with the least amount of effort.
-
No Statistical Significance
When you’re analyzing data, look beyond the summaries of key data points. Make sure that your results are actually presenting a statistical significance. Statistical significance can look a little different for every experiment, depending on what you’re testing. For example, surveys where the sample size is too small may show skewed and inaccurate results. As you test, without getting too deep into calculations and p-values, use your best judgment to decide whether something is actually valid, or if you’ve run into a case of chance.
-
Wrong Demographic Targeted
When you’re doing testing, make sure that you’re actually testing with your target audience. Hone in as much as possible on your customer persona, and find the group of people that most closely aligns with that. While it may be tempting to test with whatever audience is most accessible to you (for example friends and family), if they’re not your target audience, they may have very different reactions to your product.
As a final note, Catherine reminds us that experiments are not a one-time process. They should be run constantly, and if possible, keep tracking the outcomes of the experiments even when they’re complete. You may find that results change over time and with different demographics. Once you wrap up one experiment, you can also use those findings to start planning your next.
