Importance of Screenshot Stabilization in Visual Testing

The Visual Testing or Visual Validation technique is the most popular way to test the user interface, such as styles, UI Elements, alignments, fonts, etc. Visual testing doesn’t require advanced technical skills, the results are easy to interpret and analyze.

Visual Regression testing works by comparing two screenshots pixel-by-pixel; even if there is pixel-level change, the visual testing marks the test as failed or differences will be highlighted. The major challenge in visual testing is the stabilization of screenshots that needs a strong algorithm in the background.

Challenges faced without Screenshot / Snapshot Stabilization

1. More false failures: The false failure will be increased, and screenshot comparison becomes unstable which makes the visual validation tests worthless. 
2. Difficult to test dynamic contents: The stabilization helps to freeze gif images; though GIF image movement happens randomly, the stabilization always tries to capture the same frames as a result, visual testing will be more stable and accurate
3. Difficult to test websites having media content: Nowadays, many websites contain autoplay video features, and the video needs to freeze and be captured; otherwise, the false failure will increase.
4. More errors without custom stability options: The screenshot stabilization also means that you can identify what is causing the false failure. By writing the custom script, you should be able to make it stable. The custom option includes setting the sensitivity, changing the CSS styles, hiding the dynamic elements, etc.

Importance of Screenshot Stabilization

Screenshot stabilization is a technique or an algorithm that helps freeze the animation and other dynamic elements to reduce false failures. Let’s understand with an example the importance of it all:

We have a page with GIF image; let’s see what happens with and without the stabilization feature.

The Visual Validation Test Result – Without Snapshot Stabilization Feature

Image: Output of visual test with no screenshot stabilization

The above results from a visual testing tool that doesn’t have a screenshot stabilization feature.

• You can see that the GIF image looks more or less similar, but the visual testing tool marked the test as a fail by highlighting the difference in red.
• When you run the visual tests on the website with animation like loaders, GIFs, and videos, this is the expected outcome if the tool doesn’t support the stabilization feature.

There are tools in the market that do the screenshot stabilization before capturing the snapshot and processing the screenshot as per the given resolution, producing the output. Percy is one such tool that freezes the animations and dynamic contents and produces accurate output even if you have GIFs or animation.

Let’s run the same visual test with the Percy tool, which has a screenshot stabilization feature.

Image: Visual testing result with Percy tool

In the above image, you can see that Percy has marked No Changes, which means there are no differences. Since Percy has an in-built stabilization mechanism, it produces accurate output. The above is an illustrated example of the importance of screenshot stabilization in visual testing. 

Get Started with Percy

Customize Screenshot Stabilization in Visual Testing

Tools like Percy have in-built stabilization features; however, in some scenarios, that may not be sufficient. For example, if we have dynamic data where data changes every second (Example Date and Time), then we might have to set some custom options to stabilize the screenshots. 

1. Screenshot Stabilization for Dynamic Content

There are multiple options to stabilize the screenshots for dynamic content; for example, if you have the Date and time on the website, you can set the fake date before capturing screenshots.

Tools like Cypress provide an option to set a fake date.

// const now = new Date(2021, 3, 14) // month is 0-indexed
// cy.clock(now)

There are also other options to fake the data or response, you can use mock API or fake the responses; you can also write a custom script to replace the rendered dynamic content with a static one. 

2. Hiding Dynamic Content

The element or content changing dynamically may not be the important feature, so you can hide that completely using the hide feature of the tool you are using.

Example of Dynamic element hiding in Percy and Cypress

 cy.percySnapshot('Hide Content',{
percyCSS:`#clk_box {display:none;}`
});

You can see the difference, in the above image; Percy provides an option to hide the content while taking the snapshot. The before-hiding image shows the Clock, but the clock has disappeared after adding the code to hide dynamic content.

3. Modifying the CSS Styles

Consider you have a scenario whenever you visit the website color of some component or webpage changes dynamically. A scenario like this blocks the visual testing. In such cases, you can set the static color by using the custom CSS styling option.

Many tools provide such features; below is the example of Percy

 cy.percySnapshot('Custom Styles',{
percyCSS:`body { background: lightgrey;}`
});

The above code sets the background of the website body to light grey. Similarly, you can modify any styles which CSS supports. This helps to stabilize any dynamic content variations because of CSS.

4. Capturing Individual Elements

In some scenarios, you may not want to test the entire website; instead, you might be interested in only a single component or set of components. Testing only required components helps to reduce false failures. The advanced visual testing tool provides such options where you can specify the HTML locators to capture.

Example of Percy captures the single element

percySnapshot('Single Element', {scope:'someselector'});

5. Sensitivity Options

The sensitivity is also called the threshold, the comparison level for the screenshot. Generally, sensitivity values will be strict, recommended, and relaxed. The relaxed option compares the screenshot in a relaxed way by ignoring minor differences in the screenshots. The sensitivity option is least recommended as there is a high chance of ignoring valid visual defects.

Percy provides the sensitivity option under the Project settings.

Best Practices for Visual Testing

• Choose the tool for visual testing by analyzing the features
• Make Visual Testing mandatory while developing the user interface
• To avoid regression, execute visual tests more often
• Handle the dynamic content to avoid the false failures
• Review the visual testing results before taking further actions, as visual testing sometimes produces the false output
• Consider component-level visual testing as and when required
• Consider SAAS-based tools like Percy which provides an option to review, feedback from anywhere and helps in faster cross-team communication
• Never try to test the functionalities with Visual Testing

Closing Notes

There is no doubt that visual testing is the faster and more efficient way to test user interfaces. However, the efficiency depends on the tool you choose. The visual testing tool should produce accurate and reliable output, which should help to reduce manual verification. Screenshot stabilization is one of the important features many tools don’t provide, as this needs a strong and complicated algorithm in the backend. The screenshot stabilization helps to reduce false failures.

The custom stabilization option helps configure the webpage’s dynamic content, which the tool cannot do by default. There are many open source plugins for almost every tool however; sometimes, the outcome is not stable and reliable. A tool like Percy actively gathers user feedback and improves functionality, making tests more stable, accurate, and reliable.

Check out Percy Features