Estimate and prioritize experiments in a CRO roadmap

This article is focussed on explaining of:

  • Decide how “sensitive” an experiment should be
  • Estimate how long an experiment will take
  • Prioritize the experiments

Decide how “sensitive” an experiment shoud be

Once you decide on a hypothesis, you’ll design an experiment. How many variations to test? They are based on your expertise to generate enough data to determine the best choice. The minimum detectable effect (MDE) represents the relative minimum improvement over the original variant.

Minimum detectable effect (MDE or is also known as Minimum Detectable Lift) is a number that estimates the smallest improvement you’re willing to be able to detect over control. It determines how “sensitive” an experiment is, and in other words it’s an anticipated lift, over the control, that can be measured with a degree of certainty.

MDE is the smallest possible change that would be worth investing the time and money to design the test and implement the change permanently on the site. It’s important to know: a lower MDEs means increase traffic to ensure that a smaller lift is truly valid.

Use MDE to estimate how long an experiment will take given the following:

  • Baseline conversion rate
  • Statistical significance
  • Traffic allocation

Get the data to evaluate your hypothesis, you need to run an experiment based on a simple calculation based on MDE, visitors and precision.

Usually the parmeters to consider as standard are the following:

  • Statistical significance: 95%
  • MDE at least 20%

Using these parameters, adding number of visitors on each variants, benchmark how long to run an experiment and the impact on the business.

Estimate how long an experiment will take

The time for experiments is a crucial factor and it can be calculated using this following formula

or you can use of of these following links:

Prioritize the experiments

Prioritize experiments according to expected ROI, it’s a best-practice that it’ll help you to generate fast and easily important impact in revenue and time terms.

You can organize a simple Google Spreadsheet, adding in the vertical column the features related to prioritization of experiments: for each experiment you should consider to write the “main KPI factors”, in order to evaluate and sort the ideas through a single score number. For instance, you can consider to determine the number of weeks to run it, time for implementation and the potential business impact to calculate the sorting.

Several framework can help you to prioritize the experiments, such as P.I.E. (Potential, Importance and Ease) that it’ll help you to find which tests you should do first, calculating the medium score.

If you need a prioritization framework you can take a look at this article on Optimizely blog (Optimizely) which explains how to use a prioritization framework to evaluate your ideas.

I help Companies to Uplift E-Commerce Sales using Data Mining, CRO, Advanced Marketing Automation and Sales Funnels that works — Hyper Personalization

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

Why do you need to be data literate?

Unpicking player metrics from team performance

How to Get Better at Making Decisions and Predicting the Future

Data Analytics : An Imperative tool for Business

Summary on Cyclistic — Chapter 1: The Usage Time Difference between Casuals and Members

How Data Is Truly Important In The Sporting World & Especially In Football?

Introduction to Matplotlib — Data Visualization in Python

How to Setup Logging for your Python Notebooks in under 2 Minutes

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Alex Genovese

Alex Genovese

I help Companies to Uplift E-Commerce Sales using Data Mining, CRO, Advanced Marketing Automation and Sales Funnels that works — Hyper Personalization

More from Medium

Why Businesses fail.

“5 Ways Your Local Business Can Leverage Free Publicity To Grow” with Adrienne Dorsey & Kage Spatz

Should brands stick to sustainability?

Should brands stick to sustainability?

Does age really matter when it comes to brands?