How to write UI tests for your plugin – Introducing the Matomo Platform

This is the next post of our blog series where we introduce the capabilities of the Matomo (Piwik) platform (our previous post was How to write unit tests for your plugin). This time you’ll learn how to write UI tests in Matomo. For this tutorial you will need to have basic knowledge of JavaScript and the Matomo platform.

What is a UI test?

Some might know a UI test under the term ‘CSS test’ or ‘screenshot test’. When we speak of UI tests we mean automated tests that capture a screenshot of a URL and then compare the result with an expected image. If the images are not exactly the same the test will fail. For more information read our blog post about UI Testing.

What is a UI test good for?

We use them to test our PHP Controllers, Twig templates, CSS, and indirectly test our JavaScript. We do usually not write Unit or Integration tests for our controllers. For example we use UI tests to ensure that the installation, the login and the update process works as expected. We also have tests for most pages, reports, settings, etc. This increases the quality of our product and saves us a lot of time as it is easy to write and maintain such tests. All UI tests are executed on Travis after each commit and compared with our expected screenshots.

Getting started

In this post, we assume that you have already installed Matomo 2.11.0 or later via git, set up your development environment and created a plugin. If not, visit the Matomo Developer Zone where you’ll find the tutorial Setting up Matomo and other Guides that help you to develop a plugin.

Next you need to install the needed packages to execute UI tests.

Let’s create a UI test

We start by using the Matomo Console to create a new UI test:

./console generate:test --testtype ui

The command will ask you to enter the name of the plugin the created test should belong to. I will use the plugin name « Widgetize ». Next it will ask you for the name of the test. Here you usually enter the name of the page or report you want to test. I will use the name « WidgetizePage » in this example. There should now be a file plugins/Widgetize/tests/UI/WidgetizePage_spec.js which contains already an example to get you started easily:

describe("WidgetizePage", function () {
    var generalParams = 'idSite=1&period=day&date=2010-01-03';

    it('should load a simple page by its module and action', function (done) {
        var screenshotName = 'simplePage';
        // will save image in "processed-ui-screenshots/WidgetizePageTest_simplePage.png"

        expect.screenshot(screenshotName).to.be.capture(function (page) {
            var urlToTest = "?" + generalParams + "&module=Widgetize&action=index";
            page.load(urlToTest);
        }, done);
    });
});

What is happening here?

This example declares a new set of specs by calling the method describe(name, callback) and within that a new spec by calling the method it(description, func). Within the spec we load a URL and once loaded capture a screenshot of the whole page. The captured screenshot will be saved under the defined screenshotName. You might have noticed we write our UI tests in BDD style.

Capturing only a part of the page

It is good practice to not always capture the full page. For example many pages contain a menu and if you change that menu, all your screenshot tests would fail. To avoid this you would instead have a separate test for your menu. To capture only a part of the page simply specify a jQuery selector and call the method captureSelector instead of capture:

var contentSelector = '#selector1, .selector2 .selector3';
// Only the content of both selectors will be in visible in the captured screenshot
expect.screenshot('page_partial').to.be.captureSelector(contentSelector, function (page) {
    page.load(urlToTest);
}, done);

Hiding content

There is a known issue with sparklines that can fail tests randomly. Also version numbers or a date that changes from time to time can fail tests without actually having an error. To avoid this you can prevent elements from being visible in the captured screenshot via CSS as we add a CSS class called uiTest to the HTML element while tests are running.

.uiTest .version { visibility:hidden }

Running a test

To run the previously generated tests we will use the command tests:run-ui:

./console tests:run-ui WidgetizePage

After running the tests for the first time you will notice a new folder plugins/PLUGINNAME/tests/UI/processed-ui-screenshots in your plugin. If everything worked, there will be an image for every captured screenshot. If you’re happy with the result it is time to copy the file over to the expected-ui-screenshots folder, otherwise you have to adjust your test until you get the result you want. From now on, the newly captured screenshots will be compared with the expected images whenever you execute the tests.

Fixing a test

At some point your UI test will fail, for example due to expected CSS changes. To fix a test all you have to do is to copy the captured screenshot from the folder processed-ui-screenshots to the folder expected-ui-screenshots.

Executing the UI tests on Travis

In case you have not generated a .travis.yml file for your plugin yet you can do this by executing the following command:

./console generate:travis-yml --plugin PLUGINNAME

Next you have to activate Travis for your repository.

Advanced features

Isn’t it easy to create a UI test? We never even created a file! Of course you can accomplish even more if you want. For example you can specify a fixture to be inserted before running the tests which is useful when your plugin requires custom data. You can also control the browser as it was a human by clicking, moving the mouse, typing text, etc. If you want to discover more features have a look at our existing test cases.

If you have any feedback regarding our APIs or our guides in the Developer Zone feel free to send it to us.