How to reduce False Positives in Visual Testing?

Visual Testing or Visual Validation Testing is the best way to test the user interface of a software application. Visual validation testing is often called visual regression testing as well. It helps to catch visual errors or defects which are caused due to improper styles, alignments, and fonts. It compares two snapshots pixel-by-pixel and produces a difference as the outcome. 

Visual testing often produces false positive results that could be challenging to mitigate. In this guide, we’ll learn how to reduce glitches. 

What is a False Positive in Visual Validation Testing?

The false positive in Visual Testing results from a visual test case indicated as an error when there is no actual error present in the web application. 

Reasons for False Positives in Visual Testing

There are many false positives, such as dynamic content, animations, lazy loading, or even glitches in the webpage UI that may cause false positives. 

Some of the most common reasons for false positives in visual testing are: 

Dynamic Content

Every web application generally will have dynamic content, for example, time, date, graphs, temperature, etc. These dynamic data changes from time to time. Even though the dynamic content is displayed as expected since the visual test compares the screenshots pixel by pixel, the visual diff tool marks this as an error.

Animations

Generally, web applications contain animation effects to make UI smoother and prettier which can cause false positives during visual testing. One example is the loading icon. The loading icon continuously rotates until the page loads, there will be a minor difference in the loading icon in each millisecond as it rotates. 

So when two screenshots are compared the minor differences are marked as an error but the behavior was expected.

GIFs

There will always be movement in GIF images, when you compare the baseline screenshot with the recent screenshot you can observe some difference, as while taking the screenshots there will be a minor delay. 

Image courtesy: Tenor

In the GIF above image, the baby claps continuously. The first time when you take a screenshot it captures the below image.

Next time when you run the test there might be a small movement in the GIF image and captures that screenshot as the actual image

When the Visual Testing algorithm compares these two screenshots it finds the image as different and throws a major difference in the screenshot. As you can see in the below image, the differences are highlighted in red by the visual diff algorithm.

Autoplay Videos

Autoplay videos can also cause false positives, As the frames in the video may change every millisecond. 

How to reduce the False positives in Visual Validation Testing

The false positives make the testing unreliable. To overcome this challenge nowadays many tools are improving their algorithms. False positives can be handled to some extent in any tool. In the following sections, we will be using Percy and Cypress for code snippets as it is one of the most commonly used tools in the industry.

Consider the demo dynamic content website, where it displays some random texts dynamically. 

Courtesy: the-internet.herokuapp.com

When you run the visual tests since the contents are always dynamic the visual test always fails. There are multiple options to fix this issue.

Handling Dynamic Content False Positives in Visual Testing

Ignore/Hide the Dynamic Content

Percy provides an option to hide the dynamic content on the web page, so while performing the visual test the dynamic content will be ignored. It also hides the dynamic content by specifying the percyCSS option inside the percySnapshot() function

Percy Cypress Syntax to hide the content on the webpage:

cy.percySnapshot('<Title>',{
percyCSS:`#dynamicContentID {display:none }`
});

The above solution may not be useful in some use cases, if your website contains many dynamic contents, if you hide all the dynamic content then there will be nothing to test. In such case, you can use other alternatives

Mock (Fake) the response

There are many tools and plugins that help you to mock the request and response. Cypress, Postman, faker, etc are some examples. Using these tools you can mock the API response and convert it into static data. 

For example, if you are using Cypress you can simply mock the API request using cy.intercept() function.

cy.intercept('GET', 'http://the-internet.herokuapp.com/dynamic_content', { fixture: 'mocked_response.html' })
cy.visit("http://the-internet.herokuapp.com/dynamic_content");
cy.percySnapshot('Dynamic Data')

mocked_response.html

<!DOCTYPE html>

<!–[if IE 8]> <html class=”no-js lt-ie9″ lang=”en” > <![endif]–>

<!–[if gt IE 8]><!–> <html class=”no-js” lang=”en” > <!–<![endif]–>
<html>

<head>

<script src=”/js/vendor/298279967.js”></script>

<meta charset=”utf-8″ />

<meta name=”viewport” content=”width=device-width” />

<title>The Internet</title>

<link href=”/css/app.css” rel=”stylesheet”>

<link href=”/css/font-awesome.css” rel=”stylesheet”>

<script src=”/js/vendor/jquery-1.11.3.min.js”></script>

<script src=”/js/vendor/jquery-ui-1.11.4/jquery-ui.js”></script>

<script src=”/js/foundation/foundation.js”></script>

<script src=”/js/foundation/foundation.alerts.js”></script>

<script>

$(document).foundation();

</script>

</head>

<body>

<div class=”row”>

<div id=”flash-messages” class=”large-12 columns”>
</div>

</div>

<div class=”row”>

<a href=”https://github.com/tourdedave/the-internet”><img style=”position: absolute; top: 0; right: 0; border: 0;” src=”/img/forkme_right_green_007200.png” alt=”Fork me on GitHub”></a>

<div id=”content” class=”large-12 columns”>

<style>

.example img {

width: 82px;

height: 82px;

}

</style>
<div class=’example’>

<h3>Dynamic Content</h3>

<p>This example demonstrates the ever-evolving nature of content by loading new text and images on each page refresh.</p>

<p>To make some of the content static append <code>?with_content=static</code> or <a href=’/dynamic_content?with_content=static’>click here</a>.</p>
<hr>

</br>

</br>
<div class=’row’>

<div id=’content’ class=’large-10 columns large-centered’>
<div class=’row’>

<div class=’large-2 columns’>

<img src=’/img/avatars/Original-Facebook-Geek-Profile-Avatar-3.jpg’>

</div>

<div class=’large-10 columns’>

This is mocked static response, This doesn’t change anytime

</div>

</div></br></br>
<div class=’row’>

<div class=’large-2 columns’>

<img src=’/img/avatars/Original-Facebook-Geek-Profile-Avatar-7.jpg’>

</div>

<div class=’large-10 columns’>

Possimus necessitatibus id voluptate et consectetur rerum ipsa ut a ad hic iure ut repellat doloremque et rerum pariatur et vel cum aspernatur qui quo nostrum illo blanditiis neque sint exercitationem mollitia quam nulla dolorem.

</div>

</div></br></br>
<div class=’row’>

<div class=’large-2 columns’>

<img src=’/img/avatars/Original-Facebook-Geek-Profile-Avatar-6.jpg’>

</div>

<div class=’large-10 columns’>

Deserunt est voluptatem ea magnam est incidunt consequuntur officiis voluptas enim sed omnis error enim ab est aut vel vero pariatur harum voluptate eveniet dolor eaque suscipit voluptatem reiciendis rerum necessitatibus iusto voluptas.

</div>

</div></br></br>
</div>

</div>
</div>
</div>

</div>

<div id=’page-footer’ class=”row”>

<div class=”large-4 large-centered columns”>

<hr>

<div style=”text-align: center;”>Powered by <a target=”_blank” href=”http://elementalselenium.com/”>Elemental Selenium</a></div>

</div>

</div>

</body>

</html>

In the above code, you can see we are replacing the dynamic content API with our own static content which is stored in the mocked_response.html

When you execute the code it looks like below

Setting Inner Text using JavaScript

Many automation tools allow us to change the inner text content of the element by using simple javascript functions. This doesn’t require any complicated setup. The Javascript inner text is the best way when you have only a few dynamic elements.

For example, in Cypress, we can set the dynamic content as below

 cy.visit("http://the-internet.herokuapp.com/dynamic_content");
 cy.window().then((win) => {
win.eval('document.querySelector("div#content > div > div > div > div > div:nth-child(2)").innerText ="Some static content instead of dynamic content set by Javascript"');
});
})

Changing CSS styles

The false positive can be because of some margins, or dynamic colors, or there may be scenarios where styles are set dynamically as these are essential for the website to run smoothly but for visual testing, this will be the problem and it causes the false positive. 

Many popular tools provide the option to change the CSS styles before taking screenshots for comparison.

Example: Percy customization of styles

cy.visit("https://www.w3schools.com/");
cy.percySnapshot('Change Styles',{
percyCSS:`#search2 {background:#ff00f5}`
});

The above code changes the background color to #ff00f5 for the element having id search2 before taking the screenshot.

The above is just an example, you can do a lot more changes just by modifying the CSS styles which helps to handle the dynamic changes and reduces the false positives.

Sensitivity Settings

The sensitivity is the comparison level of screenshots, the sensitivity settings usually provide an option like strict, recommended, and relaxed. Strict mode usually checks in a detailed way. The relaxed option ignores if there are minor differences.

In Percy, you can see these options in Project Settings.

Navigate to Project Settings > Look for Advanced Options > Diff sensitivity

Note: Changing the sensitivity setting may cause the escape of the actual visual defect so, change this option only if it is required.

Closing Notes 

False positive is the most challenging and common scenario in visual testing. The application with dynamic content requires a special mechanism to handle false positives. Many tools provide options out of the box that the user just needs to configure with a few values. Percy is one such tool that provides all possible options to reduce false positives.

• It supports many frameworks which Percy terms as SDKs. 
• The support includes Cypress, Playwright, Selenium, Storybook, TestCafe, and NighwatchJS. 
• Percy provides accurate output, though it looks like just an image comparison tool, it does a lot more to make the visual diff algorithm stable and accurate.

Try Visual Testing on Percy