Description:

While working on another issue there was a UI tests where the expected image should have changed, but the UI test passed even without updating it.

Debugging through the code turned out that the problem is caused by the handling of the result of the compare command.
The error returned by the compare command is either a text error that says e.g. that image height or width differs or it is a numeric value that represents the number of pixels that differ.

The compare command of imagemagick automatically uses the precision configured for imagemagick. This is by default 6. So all numbers that have more than 6 digits will be converted to an exponential notation.

We used to use parseInt to parse the response, causing a (string) value like 1.3223e+06 to be converted to 1. As we ignore all numbers lower than 10, a test would pass even though the pixel difference was a lot higher.

@snake14 @AltamashShaikh I remember you mentioned a similar problem in the past, where we couldn't find a problem. Guess this should be fixed with this one.

Review

• Functional review done
• Potential edge cases thought about (behavior of the code with strange input, with strange internal state or possible interactions with other Matomo subsystems)
• Usability review done (is anything maybe unclear or think about anything that would cause people to reach out to support)
• Security review done
• Wording review done
• Code review done
• Tests were added if useful/possible
• Reviewed for breaking changes
• Developer changelog updated if needed
• Documentation added if needed
• Existing documentation updated if needed