Visual regression tests
We’re using BackstopJS to run visual regression tests, shortly known as VRT.
Read Leornado Giroto’s Visual Regression Testing blog post if you want to get a nice introduction.
Install BackstopJS in your repository
Create a config by running
backstop init, or copy an existing one (e.g. website or products)
Create the required visual regression test targets in the Makefile
Add the shared VRT CI file CI file to your GitLab CI pipeline
To run visual regression tests, you can use one of the obvious VRT test targets. The workflow is usually like this:
The tests are run by running
In case there’s an error the report is opened by running
Each failed test case will be inspected
For each failed test case you’ve to make one decision:
- Unwanted respectively unexpected changes:Fix the error and run the VRT tests again by running
- Wanted respectively expected changes:Approve it by either running
make vrt-approveor using the Interactive VRT report.
make vrt-approve will always approve all failed tests.
Thus using the Interactive VRT report to approve tests individually, or fixing all unwanted / unexpected changes first is highly recommended.
Interactive VRT report
BackstopJS also supports interactive web reporting, which allows you to approve tests directly in the browser, instead via CLI.
To approve tests interactively, you must run the interactive server in a separate terminal first:
The interactive server is automatically detected when you open a new report, for example by running
make vrt-report. Each failed test should then show an
Approve button in its header.
OS differences in VRT
There are OS differences, for example font renderings between Linux and macOS.
Since most of our clients are macOS, the
Visual Regression Test job of the shared VRT CI file is also running on a Mac.
Approving new reference files should therefor only happen on macOS.
Alternatively you can also trigger the CI/CD pipeline, wait until the visual regression tests failed, download the artifact and replace the corresponding files in
bitmaps_reference/ directory with the ones of the
bitmest_test/YYYYMMDD-hhmmss/ artifact directory.