How to identify which A/B test variation was the most or least successful

Cloud Auto-hébergement

Matomo A/B Testing lets you compare experiment variations and determine which performs best against your selected success metrics. Experiment reports show how each variation performs compared to the original version, including whether the differences are statistically significant.

Define what makes an experiment successful

When you create an A/B test, select one or more Success metrics to measure the performance of each variation.
ab test success metrics

The available metrics depend on your website’s configuration and may include:
* Pageviews
* Bounces
* Time on website
* Total conversions
* Total revenue
* Conversions and revenue for individual Goals
* Ecommerce orders and revenue, if Ecommerce tracking is configured

Matomo automatically lists the Goals configured for your website as available success metrics. If the Goal you want to measure is not listed, use the link in the experiment settings to Create a new goal.

Under Success conditions in the A/B test configuration, you can define the Minimum Detectable Effect (MDE). This represents the smallest improvement you want your experiment to detect. Use the default settings of 15% MDE and 95% confidence or adjust them based on the size of the improvement you want to detect and the traffic your website receives.

Compare the performance of variations

Open A/B Tests and click on your A/B test report to compare each variation against the original version. For every selected success metric, Matomo provides a summary, an evolution graph and detailed reports. This helps to evaluate how variations perform throughout the experiment.

compare ab test variations

Matomo calculates statistical significance to assess whether an observed difference is likely to reflect a genuine effect rather than random variation. As the experiment collects sufficient data, Matomo displays recommendations directly within the success metric reports. These messages can indicate when:

  • A variation performs statistically significantly worse than the original version.
  • A variation performs statistically significantly better than the original version.
  • A variation performs significantly better but has not reached the configured Minimum Detectable Effect (MDE).

For example, a variation may show a statistically significant improvement without reaching an expected MDE of 15%. This means the result is statistically significant, but the improvement has not met the defined threshold.

compare ab test variation bounces

Decide which variation to use

Review the results for your selected success metrics and consider whether the improvement meets your experiment’s objectives. A variation performing better during an experiment is not necessarily the final winner.

Before deciding which variation to use:

  • Check statistical significance: Confirm whether Matomo indicates that the difference is statistically significant, rather than a result of random variation.
  • Review the Minimum Detectable Effect (MDE): Does the improvement meets your configured threshold? A statistically significant improvement may still fall below your expected MDE. Read more about Success conditions.
  • Allow sufficient time: Run the experiment long enough to capture typical visitor behaviour, including weekday and weekend differences. You will want to ensure enough visitors have participated for Matomo to assess statistical significance.
  • Consider all success metrics: If you selected multiple metrics, check whether improvements occur in one or more areas. A successful variation would meet all defined success metrics.

For guidance on creating and managing experiments, see Create, view, track and analyse A/B tests.