Estimate and prioritize experiments in a CRO roadmap
This article is focussed on explaining of:
- Decide how “sensitive” an experiment should be
- Estimate how long an experiment will take
- Prioritize the experiments
Decide how “sensitive” an experiment shoud be
Once you decide on a hypothesis, you’ll design an experiment. How many variations to test? They are based on your expertise to generate enough data to determine the best choice. The minimum detectable effect (MDE) represents the relative minimum improvement over the original variant.
Minimum detectable effect (MDE or is also known as Minimum Detectable Lift) is a number that estimates the smallest improvement you’re willing to be able to detect over control. It determines how “sensitive” an experiment is, and in other words it’s an anticipated lift, over the control, that can be measured with a degree of certainty.
MDE is the smallest possible change that would be worth investing the time and money to design the test and implement the change permanently on the site. It’s important to know: a lower MDEs means increase traffic to ensure that a smaller lift is truly valid.
Use MDE to estimate how long an experiment will take given the following:
- Baseline conversion rate
- Statistical significance
- Traffic allocation
Get the data to evaluate your hypothesis, you need to run an experiment based on a simple calculation based on MDE, visitors and precision.
Usually the parmeters to consider as standard are the following:
- Statistical significance: 95%
- MDE at least 20%
Using these parameters, adding number of visitors on each variants, benchmark how long to run an experiment and the impact on the business.
Estimate how long an experiment will take
The time for experiments is a crucial factor and it can be calculated using this following formula
or you can use of of these following links:
Prioritize the experiments
Prioritize experiments according to expected ROI, it’s a best-practice that it’ll help you to generate fast and easily important impact in revenue and time terms.
You can organize a simple Google Spreadsheet, adding in the vertical column the features related to prioritization of experiments: for each experiment you should consider to write the “main KPI factors”, in order to evaluate and sort the ideas through a single score number. For instance, you can consider to determine the number of weeks to run it, time for implementation and the potential business impact to calculate the sorting.
Several framework can help you to prioritize the experiments, such as P.I.E. (Potential, Importance and Ease) that it’ll help you to find which tests you should do first, calculating the medium score.
If you need a prioritization framework you can take a look at this article on Optimizely blog (Optimizely) which explains how to use a prioritization framework to evaluate your ideas.
