Is it possible to change the confidence threshold / statistical significance needed to decide a winning variation?

Yes, Matomo (Piwik) uses by default a threshold of 95% but you can lower or increase the expected threshold. The higher you set the confidence threshold, the more you can be sure that the results are actually due to this experiment and not due to randomness. When you set a higher threshold, more users will be needed to participate in the experiment.