Import additional data including bots, static files, and HTTP errors tracking

By default, the log analytics importer script does not track static files (JS, CSS, images, etc.) and excludes all bot traffic.

You can enable these using the following commands:

• --enable-bots

will track search/spambots in Matomo (Piwik), using a custom variable with the name of the bot. When enabled, the log file will take longer to process since all bot page views are sent to Matomo. Matomo detects whether a log line is a from a bot by looking at the User-agent field.

Example of Custom Variables reporting Bots user agents:

• --enable-static

will specify tracking of all static files (images, JS, CSS) in Matomo. This will add some time to the general log file processing.

• --enable-http-errors

will specify tracking of HTTP errors (4xx, 5xx status) as a page view in Matomo, with a custom variable HTTP-code set to 404, 500, etc. The page title for this page view will show the URL referrer if it is specified in the log file (which can help to find out which pages have a link to a 404 for example).

• --enable-http-redirects

will track HTTP redirect (302,301,3xx) as a page view, with a custom title, and a custom variable. Note: HTTP status 304 responses (“Not modified”) are tracked as page views.

• --enable-reverse-dns

will enable the reverse DNS (used to generate the Visitors > Providers report), expect a big performance hit as reverse DNS is very slow.

• --recorders=N

specifies the number of threads: we recommend setting it to the number of CPU cores in the system (or slightly more or less depending on your server configuration)

• --recorder-max-payload-size=N

The importer uses the bulk tracking feature of Matomo to achieve greater speed. By default, 200 pageviews (or log lines) will be sent to Matomo at once. You can experiment with this number to try and achieve better performance, but there is an upper limit to the speed you can get.