Moving from CI to CD using Automated Visual Regression Testing

The past decade has seen traction in using DevOps in software testing along with the Agile Software Development philosophy. A current study by RedGate Software shows that nearly 74% of companies surveyed have adopted DevOps compared to only 45% in 2016. 

However, while software developers and non-IT stakeholders alike have welcomed this adoption of DevOps, there seems to be a disconnect in the adoption of two major components that make up DevOps, viz. Continuous Integration (CI) and Continuous Delivery. (CD).

What is Continuous Integration (CI)?

Continuous Integration allows code to be integrated to the main codebase several times a day. Each change is verified in an automated pipeline that allows for quick feedback to developers in case a bug is found.

What is Continuous Delivery (CD)?

Continuous Delivery (CD) allows teams to make changes to the codebase and push the same to production safely and sustainably. This is achieved by ensuring that every commit is ready for production and can be pushed to a suitable environment on demand. This reduces deployment time but carries the risks of hidden bugs. Thus it is imperative that thorough tests, including a strong regression test suite, are executed to check that these changes do not break the existing product.

A recent Digital Ocean report found that only 35% of the respondents have adopted Continuous Delivery as opposed to the 58% who have adopted CI. 

While there may be numerous reasons for the slow adoption of CD practices, one of the major complaints has come from the long, slow pipelines with significant wait times for testing to be completed before the code is pushed to production. 

A deeper glance into the testing process for such cases has identified the culprit to be visual regression testing.  

Regression testing is used to confirm that any changes made to the system have not affected the existing functionality before they are pushed to production. Since it is quite common for code changes to disrupt the visual aspects of the software as much as the core functionality, Visual Regression testing has acquired prime importance as well. 

Historically, this is done by comparing screenshots of the UI before and after code changes have taken place. Since this process is still largely manual for several organizations, it is extremely time-consuming and slows down the feedback loops for a CD pipeline. 

This article will look to understand better why manual visual regression tests are inadequate for modern CD pipelines and how automated visual regression testing promotes a smooth transition from CI to CD.

The present problems with Visual Regression Tests

Visual Regression tests have been a crucial component of project pipelines for a substantial period of time. However, there still exist significant challenges in the way visual regression tests are run that prevent its optimal adoption into the CD pipeline.

Delays in feedback due to reliance on manual methods 

Most visual tests are still performed manually or using poorly coupled standalone tools. The major issue arising out of these is the unnecessary delays in feedback being compiled by testers and relayed to the development team. 

This leads to a drawn-out and lengthy CD pipeline with delays in getting the codebase ready for deployment to the production environment. 

Source

Challenges in replicating visual regression tests over different environments 

Visual regression tests, like all others, must be performed on all the environments the application is expected to be hosted. This, again, is time-consuming and often needs to be done manually on different devices and system configurations before the code is production ready.  

This introduces further delays to the CD pipeline, making it an even less desirable option.

Lack of proper bug reporting for issues 

Once a bug is identified through visual regression tests, it should be documented in detail. Metadata and steps for reproducing the defect must be listed, along with the exact nature of the visual discrepancies observed.

Most tools do not offer seamless integration with bug tracking tools like JIRA and Trello, and this needs to be manually done by the developers. This adds an extra layer of complexity to this already cumbersome process and adds to the risks arising out of human error.

False positives from tools using snapshot testing 

Visual regression tests are either carried out manually or using conventional automation frameworks like Selenium and Cypress. These tools have scripts that use selectors to check the different test scenarios. The problem with this approach is that it can be altered with the evolution of the application code and lead to false negatives. This can often lead to script reworks and even the writing of entirely new scripts. This adds the burden of maintenance to this critical process, and this process can lead to significant delays for the CD pipeline by introducing unnecessary wait time.

Snapshot testing uses baseline snapshots to make comparisons during regression tests, often at the pixel level. This can lead to false positives due to :

1. Anti-aliasing: This is used to minimize image distortion. For example, rectangular pixels have jagged edges smoothed by this method. If the tests are performed on different machines with different anti-aliasing settings, it can mark the snapshots as not being in sync and raise errors due to these false positives.

Source

2. Some application components, like the number of items displayed over the cart icon or product recommendations advertised to a user, evolve over time and usage. These are also flagged as errors and need to be manually verified before discarding.

3. The use of different browsers can also lead to false positives, as images and fonts can vary depending on the rendering engine used by a particular browser.

All of these issues can lead to Visual Regression testing being a major headache while implementing a Continuous Delivery (CD) pipeline.

Leveraging Next-Gen Automated Visual Regression as the Solution

However, these challenges mentioned above are far from insurmountable. In fact, there are several automation regression testing tools that can help easily overcome these and enable the creation of CD pipelines with automation regression suites that accelerate release velocity.

Automated Regression Testing across environments

Next-Generation Automated Visual Testing tools like BrowserStack’s Percy provide an all-in-one testing, analysis, and test review platform on the cloud. Percy offers a cloud-based validation option with :

• Centralized review.
• Capturing image visual diffs parallelly.
• Cross-Browser support.
• Integration options to Azure DevOps, Bitbucket, Gitlab, Slack, etc
• Seamless integration into existing CI/CD workflows.

It also integrates with other BrowserStack offerings like Live, which provides access to a  cloud of 3000+ real devices & browsers, constantly updated and instantly accessible for remote testing. And, Automate allows the testing of a UI test suite in minutes with parallelization on a real browser and device cloud. This can allow the tests to be conducted on every commit without slowing down the CD pipeline.

Try Percy for Free

Automated Visual Regression Testing Integration with tools like Slack for fast bug reporting 

Visual bug reporting has been made more convenient when tools like Percy are integrated with Slack. This allows Percy to seamlessly communicate with communication channels like Slack to provide immediate notifications and updates to the dev team as and when the bug reports are available. The use of embedded links helps them to navigate specific builds and review them as required.

Using AI in Visual Testing to overcome false positives. AI-empowered solutions like Percy use “Visual Locators” rather than conventional selectors. These are more robust and do away with the issues arising out of hard-coded selectors.

AI in visual regression testing locates the elements by “vision,” just like in the human world, and is able to correctly identify the changes using visual AI. This is a step above normal computer vision and can analyze and learn from experience while processing visual information. This leads to an almost “human” level of fidelity and confidence when checking for visual changes to the UI.

All of this leads to a shorter, wider CD pipeline that allows teams and managers to be flexible in deciding when to execute which particular suite of tests based on the change made and be production ready at a moment’s notice.

Source

Conclusion

The State of Visual Testing report indicates that:

• 58% of the respondents have adopted some form of visual testing, and it is seen to be both scalable and practical from a front-end testing viewpoint.
• 47% of respondents use visual comparison tools and have witnessed a significant acceleration in visual testing adoption due to the results obtained.

The use of appropriate automated visual regression tools like Percy can lead to a major speed-up of the Turnaround Time for visual regression tests in the CD pipeline leading to shorter feedback loops and a greater degree of confidence in the builds being certified for release.

Testers can leverage these tools to increase visual coverage across the UI and eliminate the risk of post-production bugs. They can also avoid false positives and get real-time results with reliable regression testing. This allows them to release software faster with DOM snapshots and parallelization techniques to quickly process complex test suites in a scalable manner.

Run Visual Regression Tests on Percy