In at least 3 separate plugins (FormAnalytics, AbTesting, and Cloud), we have noticed that some of the UI test cases return false positives. The test cases are passing despite the processed and expected screenshots having significant differences. As this is affecting multiple plugins and it's happening in local and build environments, this seems more likely to be a core/library issue. If there's a change that we can make in the plugins, that would be great, but we haven't been able to find one yet.

Context

We've had cases where the UI tests didn't alert us to a regression because the test cases continued to pass despite the screenshots not matching. The most recent issue was when I made a change to one of the AbTesting screens, but the build still passed without me updating the screenshot. A couple weeks ago, the Cloud plugin had a couple screenshots passing despite the background color being different for the entire screen. Back in April (build 4685686384), I noticed that one of the FormAnalytics test cases was still passing despite the processed screenshot being practically empty. You can find some conversation about it in the development channel on 19 April.

Expected Behavior

I expect the UI builds to pass when the screenshots match and fail when they don't.

Current Behavior

The UI builds pass and fail as expected most of the time, but sometimes they pass even when there are significant differences between the screenshots.

Possible Solution

Not sure. Change image comparison libraries?

Steps to Reproduce (for Bugs)

1. Try changing something obvious in AbTesting/vue/src/Experiments/Edit/Embed.vue
2. Run the UI tests locally or commit and push to run them via Github Action
3. Note that the UI build still passes despite the screenshots no longer matching

Your Environment

• Matomo Version: 4.15.1 or 5.0.0-rc2 currently, but it's been an issue for months
• PHP Version: 8.1, 8.2, probably others
• Server Operating System: Ubuntu and whatever the builds use
• Additionally installed plugins: Pretty much everything but the stock ones are disabled