How do I run the Log File importer script with default options?

If you are using Matomo On-Premise, then you find the importer script in misc/log-analytics/import_logs.py.

If you are using Matomo Cloud or Matomo for WordPress, you can download the script from GitHub.

$ python /path/to/matomo/misc/log-analytics/import_logs.py

This will display the help information. The only required parameter is

--url=https://analytics.example.com

To specify the Matomo (Piwik) base URL. Then, you can specify one or many log files to import.

If you run the script on a server other than your Matomo server, then you will also need to specify the --token-auth=SECRET_TOKEN_HERE parameter (learn more in our FAQ: where do I find token_auth?).

There are many more options available. See the help output, and the README for more information and explanations about available parameters.

For example, if you wish to track all requests (static files, bot requests, http errors, http redirects) the following command would be used:

python /path/to/matomo/misc/log-analytics/import_logs.py --url=https://analytics.example.com 
--idsite=1234 --recorders=4 --enable-http-errors --enable-http-redirects --enable-static
--enable-bots  /path/to/your/web/access.log

User

The Python script needs to run as a user with correct permission. If you get a silent fail during the import (0 requests imported successfully), try changing the user. During testing, the easiest way to do this may be to run the script as superuser (« sudo python3… »). We recommend to set a more specific user if you are going to run this process routinely.

Input file format

The Python script will compare the lines of your import file until it finds a match with one of its known formats.
You can see the list of available formats if you inspect the script of the import_logs.py file and search for FORMATS.
The exact names of the formats are currently: ‘common’, ‘common_vhost’, ‘ncsa_extended’, ‘common_complete’, ‘w3c_extended’, ‘amazon_cloudfront’, ‘iis’, ‘shoutcast’, ‘s3’, ‘icecast2’, ‘elb’, ‘nginx_json’, ‘ovh.

Common options

You can manually set the format for parsing the ingested log:

--log-format-name ncsa_extended

You can troubleshoot the log-file-format selection process:

--debug

Important option: there is no check for redundant data – the log importer will double or triple the data at its destination without giving any warning, so always import to a test site first. You can choose a destination site ID with this option:

--idsite=1234

Complete example for Linux Apache

If you are using a log rotation system, and your server is for example Apache, you will need to first search for which log to import, inspect the file dates, and then add that path to the import script.

Putting all of that together, here is what your commands might resemble:

sudo ls /var/log/apache2/
sudo head /var/log/apache2/access.log.1
sudo head /var/log/apache2/access.log.9
sudo tail /var/log/apache2/access.log.9
sudo python3 /path/to/matomo/misc/log-analytics/import_logs.py --url=https://analytics.example.com --idsite=1234 --enable-http-errors --enable-http-redirects --enable-static   --token-auth=REDACTED --debug --log-format-name ncsa_extended  /var/log/apache2/access.log.9  

Learn more about how to use the script, how to import logs automatically every day, and advanced setups in our Log Analytics Readme.

If you have feature requests for better Server Log processing with Matomo, please let us know using the feedback form below. We look forward to your feedback and hope Matomo will deliver huge value for all server logs.