How AI in Visual Testing is transforming the Testing Landscape

The distinction between Computer Vision & Visual AI

Computer vision involves training a machine to process visual information in a way similar to human sight. Human sight involves using retinas, optic nerves, and the visual cortex to give context to images, such as whether a car is moving or not. Computer vision utilizes cameras, data, and algorithms to carry out the same function but in a more efficient manner. A machine can be trained to analyze thousands of images in a second in a more accurate manner than human vision.

Computer vision enables computers and systems to derive meaningful information from digital images, videos, and other visual inputs and take actions or make recommendations based on that information. However, Computer vision is not the same as AI in Visual. Where artificial intelligence allows a computer to form thoughts, computer vision enables a computer to see and process visual information. Visual AI needs to be employed to visually analyze, learn from experience, and emulate human intelligence while processing visual information.

Applications of Computer Vision and AI in Visual Testing

Computer Vision and Visual AI are two of the most rapidly evolving technologies, their uses and effects on everyday life are innumerable. To provide context to the immense potential of these technologies let’s explore some of their applications.

Image classification is one of the most important applications of these two technologies. Image classification is the ability to see an image and accurately discern that it belongs to a certain class.

An example to illustrate the use of such classification algorithms would be Optical Character Recognition (OCR). OCR is a computer vision task that is widely used in our everyday lives. OCR carries out text recognition, and it is commonly used in order to digitize paper-based documents and make them machine-readable so that further analysis/work can be easily carried out. For example, Medical notes and reports can be digitized so that patient data can be easily extracted for further analysis of disease rates.

Similarly, object detection is another important application. Object detection uses image classification algorithms to learn to identify a certain class of images. It then learns to detect the visual of that image in other images or videos.

Object tracking is a deep learning process, which takes object detection a step further. This process involves training a machine to start following or tracking the object once detected. For example, modern phone cameras are highly adept at tracking and focusing on human faces.

Due to the diverse and creative ways these technologies can be used, as illustrated above, they are at the forefront of driving improvements and growth across various industries.

How AI in Visual Testing is Revolutionizing the Testing Landscape

Traditional testing involves a human manually checking tools for visual changes, or writing test scripts using automation frameworks such as Selenium. These scripts use selectors and need to carry out and check for various test cases. The problem with using selectors however is that they can be altered as the application code evolves. This can lead to false negatives and require the developer to rework and update their test scripts frequently. Sometimes an altogether new test script will have to be written. This frequent debugging and rewriting get iterative and often leads to delays.

AI provides a better alternative. Several testing tools use AI in Visual Testing and currently employ “Visual Locators” which are more robust and eliminate the issues caused by hard-coding with selectors. Since AI in Visual Testing is used to locate the elements by vision, much like a human would, even when the selectors for the element change the AI is able to visually identify them.

Snapshot Testing and Its Limitations

Snapshot testing is popularly utilized to test the cosmetics of an application. This testing is carried out in order to detect any visual changes in the application and ensure that visual regression has not occurred.

However, there are limitations to this method.

The essence of snapshot testing is that there are baseline snapshots against which the testing tool carries out comparisons, often at the pixel level. This leads to several false positives due to the following reasons:

1. Anti-Aliasing is used to minimize the distortion of images. Rectangular pixels often create jagged edges which can be smoothed and rounded using anti-aliasing. The settings for anti-aliasing can be changed, and if snapshot testing is done on machines with differing settings for anti-aliasing all the snapshots being compared would be tagged as changed.
2. Certain fields of an application are meant to change over time such as the number of items displayed in a bubble over the shopping cart icon, or recommendations advertised based on user preferences. These would also be flagged as changes when they should be ignored.
3. Using different browsers can also lead to false positives as images and fonts can vary from browser to browser depending on the rendering engine used by the browser in question.

Due to the aforementioned reasons, snapshot testing is not very popular with QA test engineers since it leads to a large number of false positives to sift through manually.

While AI in Visual Testing has greatly advanced the visual regression testing landscape, there still remains a dire need for tools, which are able to use AI in Visual testing to carry out cosmetic testing with more sophisticated change detection.