What is A/B testing?

An A/B test is a method of comparing different versions of your websites or apps, and see which variation makes you more successful.
A/B tests are also known as experiments or split tests. In an A/B test you show two or more different variations to your users (visitors) and the variation that performs better wins. When a user enters the experiment a variation will be randomly chosen and the user will see this variation for all subsequent visits. Experimenting in this way lets you maximise your success. Read more about A/B Testing on the user guide.

Where can I get A/B Testing?

A/B Testing is a plugin for Matomo and is available for purchase on the Matomo Marketplace as a yearly subscription. While the subscription is active you will receive all updates for this plugin.

You can also get it as a hosted solution on our Matomo Analytics (formerly Piwik Analytics) Cloud.

Where do I get more information about this plugin?

We recommend to visit the A/B Testing website as well as the A/B Testing plugin page on the Matomo (Piwik) Marketplace as a starting point. Developer documentation can be found on our Developer Zone.

Who develops & maintains the A/B Testing plugin?

The plugin is developed and maintained by InnoCraft, the company from the makers of Matomo. At InnoCraft, talented and passionate developers build and maintain the free and open source Matomo Analytics (formerly Piwik Analytics) platform. This ensures the highest quality and compatibility for all of their plugins. As a result, popular features like Segmentation work out of the box with this plugin.

What about the quality of this plugin?

This plugin is built and maintained by InnoCraft, the makers of Matomo (Piwik). Therefore we can make sure the plugin is well integrated, kept up to date and automatically tested whenever a change is made in Matomo. By purchasing this plugin you also help the developers to being able to maintain the free and open source project Matomo itself.

Does A/B Testing work as a standalone product?

No, A/B Testing is a plugin that is built on top of Matomo (Piwik). To use it, you first need to install Matomo. Matomo requires PHP, a MySQL database and a webserver like Apache or Nginx. Learn more about Matomo. Alternatively, to take away the hassle of installing and maintaining your own Matomo, signup to our Matomo Cloud (formerly Piwik Cloud) service.

Which Matomo version is required for this plugin?

Matomo (Piwik) 2.17.0 or newer is required. You can also signup to our Matomo Cloud (formerly Piwik Cloud) service.

What are examples of A/B tests?

Here are just a few examples of typical A/B tests experiment that can increase your website’s revenue and conversion and grow your business:

  • Often your goal is to convert people to submit a form to contact you, or signup to your service. Experiment with different form designs for example a simpler form asking for less information, making the form fields bigger and more visible, or changing the labels of your form fields.
  • In your marketing landing pages you may display a photo or several photos. Experiment with other photos for example: a photo showing a happy person vs a photo showing your products in action vs several smaller photos.
  • in your marketing pages or marketing campaigns, experiment with using different headlines and call to actions text.
  • Experiment with a different navigation on your website, by reorganising your menu or even re-designing your menu. Compare your success metrics for a vertical menu vs horizontal menu.
  • On your ecommerce website, you can test the label of your Call to action button: Add to Cart vs Buy Now vs Purchase vs Checkout.
  • Experiment with the location of your button on the page.
  • Experiment with the style of the button: colors, size, …
  • In your checkout flow, experiment with removing some elements of your page (selectors, buttons, links) to make your page as simple as possible and compare with your original design to see whether your ecommerce orders increase.

Happy experimenting!

I do not have any experience with A/B testing yet, will I still be able to use it?

Yes, one of our main goals from the beginning was to explain each individual step of setting up an experiment directly in Matomo:

  • We show inline help within the UI
  • We guide you through the configuration of an experiment
  • We help you analyze the report by providing conclusions based on the tracked data
  • We show useful information about the experiment directly in the report to remind you what you wanted to achieve initially

Apart from this, you need a developer to embed A/B tests into your project. The A/B testing JavaScript framework requires little knowledge about JavaScript and lets you run experiments quickly. You can also run A/B tests on servers, apps and in campaigns.

Which browsers are supported by the Matomo JavaScript A/B Testing framework?

We support the same browsers as Matomo (Piwik) does, meaning experiments will run even on older browsers.

Which browsers are supported to manage experiments?

We mainly use Matomo (Piwik) UI components provided by the Matomo platform. Therefore we support the same browsers as Matomo does.

We are already using an A/B Testing framework, can we keep using it?

In many cases this will be possible. This plugin was built to be usable with most A/B testing frameworks for JavaScript, Ruby, Java, Android, iOS, Python… Some popular A/B test libraries are for example PlanOut by Facebook (Java, PHP, JavaScript, Go, Ruby), phpab (PHP) or Vanity (Ruby).

When you use a third-party A/B testing framework, all you need to do is letting Matomo (Piwik) know which variation was chosen by tracking an event or by specifying a JavaScript variable in your HTML. Read more about the different options to run your experiments.

Is any of the tracked data shared with another party?

No, the data will be only tracked into your Matomo (Piwik). 100% data ownership and privacy.

Where can I create, edit or manage my A/B test experiments in Matomo?

To manage your experiments:

  • in Matomo (Piwik) 2 log in to Matomo and click on « Personal => Experiments ».
  • In Matomo 3 you can find the menu item under « Administration => Experiments ».

Can I perform redirects to compare entirely different pages or layouts?

In case you want to experiment and compare the performance of different pages URLs, you can create your experiment where you specify a Redirect Page URL for each variation.

When you create your experiment, under the section « Redirects », for each variation (including the « Original » variation) you can can enter the Page URL to redirect to and test.

Here is how it works: when users land on the original page URL, they will enter the experiment (and be assigned one of the Redirect URL variation). Then they will either 1) stay on the current page (if the Original variation was picked), or 2) be automatically redirected to the Redirect URL they were assigned.

When you create an experiment using « Redirect URLs », please make sure to include the Matomo Tracking code on all your pages including on the Redirect URLs pages.

For more information and ways to test different pages and use redirects in A/B Tests.

How do I decide which variation within my experiment is the most successful?

When you configure an experiment, Matomo (Piwik) lets you define which outcome will be considered a success to you. You can choose success conditions and success metrics. Success conditions let you for example define how big of an improvement you expect from your A/B test (the Minimum Detectable Effect).

Some of the available success metrics include:

  • Goal conversions
  • Goal revenue
  • Ecommerce orders
  • Ecommerce order revenue
  • Total number of conversions
  • Total revenue
  • Page views
  • Bounces
  • Visit length (time)

You can select any pre-existing goal as a success metric, or create as many new goals as you wish. For each selected success metric you will see a summary and an evolution graph across all variations as well as individual reports for each success metric.

How do I know which of my variation was the most or least successful?

Matomo (Piwik) will show you reports for each selected success metric to see how the different variations compare against each other. At a glance you will be able to see whether a variation is currently performing better or worse compared to the original version.

Matomo also shows you how confident we are that the improvement is actually due to this change (statistical significance) and for how long the experiment should run at minimum.

As soon as enough users have entered an experiment, Matomo will start giving you recommendations about the results.

  • When a variation is improving your success metric, it will be highlighted as the winning variation.
  • When a variation is statistically significantly making your success metrics worse, it will be highlighted as a losing variation.

How do I embed an experiment into my project?

When you create an experiment, Matomo (Piwik) shows you the code that needs to be embedded into your project in order to run the experiment. The implementation depends on where you want to run an experiment:

Can I run an experiment only on a subset of my users?

Yes, you can allocate only a certain percentage of your users to enter an experiment. For example you can configure that only 20% of your users should take part in an experiment.

It is also possible to define a custom JavaScript trigger method to further restrict who enters an experiment. For example you could activate the experiment only for users from a certain location. Read more about this in the developer docs for running an experiment.

Can I change the traffic allocation between variations?

Yes, you can fully customize the ratio of how often a certain variation should be selected compared to your other variations.

Say you have three variations « Blue », « Red » and « Green » plus the « Original » version. By default, Matomo (Piwik) allocates 25% of your traffic to each variation. You may also choose to allocate more traffic to « Blue » and « Red » while your « Green » and the « Original » version gets less traffic.

Is it possible to run experiments on the server, for example using PHP, Python, Java or any other language?

Yes, it is possible: you can do this by using any A/B testing framework of your choice. Read more about this in the guide for running experiments server-side.

I want to run an experiment that needs changes on the server and in JavaScript, is it possible?

Yes, it is. The workflow would be to first activate any of the experiment variations on the server and then forcing the same variation in the client.

Is it possible to run experiments on an Android or iOS mobile app?

Yes, it is possible in combination with the Matomo (Piwik) Android SDK or Matomo iOS SDK and an A/B testing framework that is compatible with your mobile platform. Read more about this in the guide for running experiments in an app.

How do I run an experiment in a marketing campaign such as a banner ad, search marketing ad or an email marketing campaign?

For email campaigns your newsletter tool needs to support A/B tests (assigning your recipients into different groups). You can then send different newsletters to your visitors and see how the different newsletters affect the browsing behaviour on your website. This is only recommended when you have many users that click on your newsletter.

To track how different ad or email campaigns affect the browsing behaviour on your website, you only need to add two URL parameters to your campaign URLs. Read more about this in the guide for running experiments in a campaign.

When do I typically need A/B tests?

Here are just a few use cases you may experience in your day to day work.

Redesign your most important landing page

One of your most important landing page has been redesigned to better showcase the value of your services, and you need to make sure the new version is not making things worse. To compare the performance of two new versions and check the new version does not decrease number of contact form signups or the , you run a A/B test and see whether the redesigned version is converting more visitors to reach your goals.

Optimise your conversion funnel

As a marketer or product manager, you often need to improve conversions at any particular step of the conversion funnel. Basically you need more visitors to get closer to the end of the funnel and convert your goals (whether signing up, leaving an email address, downloading a brochure, watching a video, etc.). To improve your conversions you may start by identifying the bottlenecks and what is causing users to leave the funnel and where. For example, you may be losing many users at a particular step of your conversion journey such as your signup form or your checkout page. You and your team may then brainstorm several solutions that would make things better, and your team decides to implement the most important change first. When you implement this change (and any others) using A/B Testing, you are actually now scientifically measuring and the A/B testing product will confirm (or deny) that your change has indeed improved conversions (and how much improvement).

Soft launch and test your new feature (advanced)

As a product or game designer, you are building an app, game or any other software product, and you wish to soft launch a new exciting and innovative feature. Soft Launching is a great way to get ideas tested early without much risk. Soft launching means that your new feature will be visible only on a subset of users and compare behaviour. So you can use A/B Testing to soft launch your apps and games features, and measure how the new feature performs on for example 10% of your user base. A/B Testing indicates whether users are better engaged in the app (across all your success metrics), after using the new feature.

See more example in the blog post: A/B Testing Platform for Matomo Analytics (formerly Piwik Analytics).

Is it possible to segment Matomo reports by A/B test experiments and variations?

Yes, you can segment your visitors by experiment they have participated in, and even segment your visitors by each specific variation they got to see. This lets you get more insights into how t different variations have affected your other reports in Matomo (Piwik).

Say you are running an experiment « Buy Now Button Color ». You can segment Matomo reports by visitors that have seen the « Blue color » of the « Buy Now Button Color » and compare the report to the visitors that have seen the « Red color ». This lets you compare in detail how each variation has performed against each other.

By using the segmentation in combination and viewing the Visitor Log, you can see in detail the path and actions of each of your users who viewed a particular variation. This lets you view how the behavior of a particular user may change depending on which variations they got to see.

Note: if you want to segment and view only users who entered your Original variation, it is required to write the name in lowercase as follows: original. The segment editor will look like this: « Variation is original ».

Is it possible to apply other segments (such as Country or Marketing campaign) to the A/B tests reports?

Yes, you can apply any Matomo (Piwik) segments to your A/B tests reports and see for example how your different variations compare against each other in different countries (when applying the « Country » segment), or how your variations compare depending on which marketing campaign or referrer brought the visitor to your website or app. To do this open an A/B test report, then create or apply your custom segment.

Is it possible to schedule an experiment to start and finish at a certain date and time?

Yes, you can optionally schedule a start and finish date for your experiments. To schedule a starting or ending date manually, edit your experiment and click on « Schedule ». By default, an experiment starts automatically as soon as the first user enters your experiment, and the experiment can be marked as finished at any time.

For how long should I run an A/B test experiment?

The time it takes for an A/B test experiment to run depends on the experiment, your business cycles, the improvement rate you expect and many other factors such as conversion rates. Once an experiment is running, Matomo (Piwik) will let you know the minimum number of visitors that need to participate in your experiment in order to reach a statistically significant result that will be repeatable and not due to random chance. The amount of needed visitors may change while the experiment is running.

We recommend to let experiments run for several business cycles, for example several full days or even better let the experiment run for several full weeks, as user behaviour changes at night and on the weekends.

Is it possible to change the confidence threshold / statistical significance needed to decide a winning variation?

Yes, Matomo (Piwik) uses by default a threshold of 95% but you can lower or increase the expected threshold. The higher you set the confidence threshold, the more you can be sure that the results are actually due to this experiment and not due to randomness. When you set a higher threshold, more users will be needed to participate in the experiment.

Is it possible to define on which webpages an experiment should be activated?

Yes, by default an experiment is activated on all of your web pages. You can also choose to customize on which pages an experiment should be activated. For example you can define to activate an experiment only if the path of an URL starts with « /blog » and has a URL parameter « language ». There is also a tool in the Experiments creation wizard which lets you test whether users would enter an experiment on a given URL or not.

How is chosen which variation a user gets to see? Is a request to the Matomo server needed?

No request to the Matomo (Piwik) server is needed in order to pick which variation a user will see.When a user enters your experiment for the first time, a variation is randomly chosen within the browser (JavaScript). On subsequent visits, the visitor will get to see the same variation as we persist the name of the chosen variation in the browser.

How do we ensure a user always sees the same variation when visiting the website multiple times?

By default, we try to store the activated variation name in the Local Storage if the browser supports this feature. If the browser does not support this feature, we will set a cookie for up to one year to remember which variation was once selected for this particular visitor.

How do I test a specific variation or share a URL for a specific variation?

You can add a URL parameter ?pk_ab_test=$variationName to any URL to force the activation of a specific variation. Make sure to replace $variationName with the name of a variation. If you run multiple tests on the same page, you can activate different variations by specifying the variation names comma separated: $variationName1,$variationName2.

This is useful when you are integrating an experiment and you want to test each variation or when you need to share a URL for each variation with your colleagues to get an approval before running an experiment.

Can I create experiments via the Matomo HTTP Reporting API?

Yes, you can create, update, finish and archive experiments via the Matomo (Piwik) HTTP Reporting API.

Can I fetch experiment reports via the Matomo HTTP Reporting API?

Yes, you can request experiment reports in different formats such as JSON, XML, CSV and HTML to integrate the results anywhere. Read more about the Matomo (Piwik) HTTP Reporting API and view the Reporting API Reference.

Do I get access to the raw experiments data that was tracked?

Yes, if you host Matomo (Piwik) yourself you get access to all data that is stored in your MySQL database. The tracked data is stored in a table called matomo_log_abtesting, the A/B test configurations are stored in a table matomo_experiments.

Under what license is the JavaScript A/B testing framework released?

The Ab testing plugin and the JavaScript A/B testing framework are released under the InnoCraft EULA.

How do I monitor changes to A/B tests experiments?

The Ab testing plugin integrates with the Activity log plugin to let you see who and when changes were made to your A/B tests, for example when a given experiment was started, finished or archived.

Does the A/B testing framework add a lot of data to the Matomo JavaScript tracker?

No, the A/B testing JavaScript framework only adds a few kilobytes to your Matomo (Piwik) JavaScript tracker piwik.js. Having a small footprint and minimizing the tracker size is very important to us.

How do I exclude employees or partners views and clicks from affecting my A/B tests results?

See our FAQ: Excluding traffic from Matomo (Piwik) using IP Addresses, Cookies, and more.

Is it possible to create an unlimited number of A/B test?

Yes, you can create as many A/B tests (experiments). With Matomo (Piwik), our goal is to provide a no data limit policy. We recommend to:

  • run only a few experiments at the same time as they may affect each other.
  • or run only experiments which do not affect each other.

Is it possible to use an unlimited number of variations within an A/B test?

Yes, it is possible to compare an unlimited number of different variations. In a typical A/B test, two variations A and B are compared against each other.

You may however create more variations as you wish. However please note that when testing many variations at the same time, the time it takes for Matomo (Piwik) to reach a statistically significant result (and decide which variation performs better or worse) will increase with the number of variation you are testing. This is why we recommend to test only a few variations at the same time.