How CI CD Tools have revolutionised Automation Testing?

The turn of the century has led to several tech buzzwords making their mark in the world of software development. From Net Neutrality to Big Data, AR/VR to Blockchain, the past vicennium has been chock-a-block with proverbial game changers making their entrance to the IT world. Of these, DevOps, coined in 2009 by Patrick Dubois, has firmly established itself as the gold standard for streamlining software development processes to ensure smoother delivery cycles.

A 2021 study by Redgate Software indicates that 74% of all companies surveyed have adopted DevOps as a standard software development practice compared to only 45% back in 2016. 

And, at the heart of the DevOps process lies CI/CD (Continuous Integration and Continuous Delivery). This methodology seeks to bridge the gap between Developers, Testers, and Operations by continuously integrating the development work done, testing it rigorously (Continuous Testing) to ensure the necessary standards are met, and leveraging automated pipelines to allow teams to release products and updates with greater speed and consistency. 

The Evolution of Automation testing

The practice of test automation can be defined as one in which automation tools/frameworks are leveraged to execute a suite of tests without the necessity of a human agent to oversee the process. Test Automation can be defined as a practice in software testing that leverages automation tools or test automation frameworks to manage the execution of tests instead of a human tester. It compares actual test results with expected results translated as assertions inside of our test scripts. As a result, automated Testing provides great effectiveness and faster results.

At the start of the century, testing was largely confined to spreadsheets and test management tools like HP QC and other in-house options. There were few options for automation testers and tools by SQA, Mercury, and Segue dominated the market. 

The advent of Selenium around 2004 was a definitive game-changer and made automation testing step from the shadows of proprietary tools into the open-source space. Cloud testing tools like BrowserStack Automate supports the running of Selenium tests using multiple frameworks across 3000+ real browsers and devices. It is recommended to test on real devices taking real user conditions into account to get more accurate test results.

In traditional software development, testing was relegated to the end of the cycle, and the need for automation testing was primarily for edge cases and regression rather than the norm. 

However, with the advent of Agile testing and DevOps the need to test each and every bit of code before merge and release became of the utmost importance. It was no longer sufficient to cluster all testing processes towards the end, and conventional manual testing methods were too slow and too cumbersome to effectively deliver bug-free code in the tight timelines required by Sprint planning. 

The rapidly changing landscape required a shift to a continuous delivery model where software was continuously in development and needed to be deployed continuously. This necessitated continuous testing to ensure quality control at all stages. 

Test automation was crucial to this model as:

• It helped save time by allowing speedier execution of test cases
• Optimised test coverage in terms of speedier coverage of functionalities 
• Several data-driven, and regression activities could be carried out in parallel
• It freed up manual testers to carry out critical activities like exploratory testing, and test planning and served as an efficient ally for improving ROI

Testing in Context of CI/CD Pipeline

However, in order for the CI/CD process to work as per customer expectations, a number of business requirements need to be satisfied. In some cases, these might be either a suite of system tests for an individual body of code whereas in others it may be a set of UI tests, while others still might require a more complex mix of integrated tests to be run. 

Regardless of their complexity, these can be visualised into a set of steps executed in series as well as in parallel. In the jargon of DevOps, each step is called a stage, and the ordered set of these stages is known as a pipeline. 

Source

Though the specific stages of the pipeline can vary depending on business needs, any pipeline will be considered to have been executed when a trigger (e.g a commit) has been activated. One this execution starts, the stages are executed in order of precedence. When a set of parallel stages are reached, the stages are executed simultaneously. 

DevOps leverages the power of test automation in a number of different ways.

• It can incorporate automated test suites into CI/CD tools-based pipelines to test pieces of code before integrating them into the main body. 
• It can leverage parallel test tools to run these tests in parallel to speed up individual stages. 
• And it can even perform automated visual testing to ensure the UI Design is consistent as per the user requirements 

The use of test automation not only speeds up the go to market time, but eliminates the risk of manual errors and brings a sense of consistency to continuous testing. 

To fully appreciate the revolution in terms of the usage of automated tests for the CI/CD pipeline, the following verticals can be considered:

Incorporate Automation for Cross-Browser Testing

Companies that develop web-apps need to be careful about ensuring that their sites and features work seamlessly on all browsers and systems. If a user working off a browser or OS on which the system has not been tested finds issues with the functionality or the responsiveness, at best they will feel this is a temporary issue and return later.

At worst, they will leave forever to find a better solution. In either case, the company loses out on revenue and customer traffic. Browser windows, though all calibrated to Open Web Standards, have their own interpretations of them. They each render HTML, CSS, and JavaScript in unique ways, thoroughly debugging the website or app’s source code is not enough to ensure that it will look and behave as intended on different browsers (or different versions of a single browser).

For cases like this, teams depend on cross-browser testing to help pinpoint browser-specific compatibility errors so that they can be debugged quickly.

Cross browser testing encapsulates a wide range of activities including –

• Base functionality tests 
• Visual testing for checking the UI Design
• Accessibility testing to check compliance with Web Content Accessibility Guidelines (WCAG)
• Responsiveness testing

Run Cross Browser Tests on Real Devices

Though emulators or simulators can be used these results are not easily scalable and have been shown to be unreliable on virtual mobile platforms (Android and iOS). Moreover, they do not take any real user conditions into account while testing, thus, the chances of leaving bugs are higher.

Thus, cross-browser testing tools like BrowserStack Live have come in which allow teams to efficiently test their apps on different scenarios before finally pushing the code to the production environment. Not only does this lower infrastructure costs, but many of them are low-code or no-code systems for which you don’t have to know coding to make use of interactive cross browser testing tools. 

Debugging Toolkits for Defect Reporting

A major roadblock in terms of continuous delivery is the scenario where the lack of proper communication between the QA Team and developers leads to non-reproducible bugs being pushed to production. Automated testing coupled with bespoke debugging tools can help enable the effective identification, reporting, and resolution of defects across systems.

BrowserStack’s debugging toolkit makes it possible to easily verify, debug and fix different aspects of software quality, from UI functionality and usability to performance and network consumption. Through its text logs, video logs, console logs, network logs, and telemetry logs debugging becomes easy as the bugs can be well reported via channels like Slack, JIRA, etc.

Parallel Testing to reduce Test Times

The USP of automation testing lies in the fact that these tests can be executed at a speed far greater than what is possible by a manual tester. However, sometimes the constraints of the Sprints and the effect of Wait Time on the CI/CD pipeline means that sequential automation of test cases is too slow to be effective

In this, CI/CD tools have brought to the forefront the need of parallel testing. This allows testers to simultaneously run multiple tests on multiple devices, and cut down test execution time, expedite result generation, and enable continuous testing to occur within shorter timelines. 

A simple example can help explain how parallel testing can boost automation test capabilities. 

If an automated functional test of a signup form takes an average of 2 mins for 45 different Browser/OS configurations then the total run time is 90 mins or 1.5 hrs when run successfully in sequence. If three tests are run in parallel, this can be reduced to 30 mins

And for six tests running in parallel, the run time would be a measly 15 mins.

For more complex scenarios involving a shift in any of these parameters, it is best to use a Parallel Test Calculator to understand the number of parallel sessions required to achieve test coverage and build execution time goals.

With Parallel Testing, BrowserStack Automate and App Automate allow you to run multiple tests in parallel across various browsers/devices and OS combinations. In this way, more tests can be run at a time, thereby decreasing the overall time spent on testing.

Automate Visual Testing for Faster Releases

Visual testing or Ui testing helps to verify that the UI is rendered appropriately for all users. Visual tests

1. check that each element on a web page appears in the right shape, size, and position
2. checks that these elements appear and function perfectly on a variety of devices and browsers.

Previously, this used to be the sole purview of manual testers and was deemed a time and cost intensive process. However, with the advent of CI/CD this has been incorporated into the testing pipeline and test runners have been created to write and run tests. Each test carries assertions defining a condition that will either pass or fail. 

Once tests are written, use a tool like Selenium or Cypress to interact with the browser. They support visual tests and allow for the generation of screenshots of web pages.

Finally, there must be a tool for managing the testing process. Percy by BrowserStack is one of the best-known tools for automating visual testing. It captures screenshots, compares them against the baseline images, and highlights visual changes. With increased visual coverage, teams can deploy code changes with confidence with every commit.

Improve Quality Control with Automation Test Frameworks

CI/CD tools have not only allowed automation testing to evolve into a core part of the software delivery process, but they have also allowed integrations with multiple test automation tools and frameworks. 

Jenkins, one of the most popular CI orchestration tools provides plugins for:

1. Running automation test suites: Jenkins provides plugins for test frameworks like Selenium, Cucumber, Appium, and Robot frameworks and gathers the results for presentation into dashboards with detailed reports on test failures. 
2. Rendering Live Trends: This helps to tabulate test results, and log failures concurrently.This is done using a trend graph which shows the comparative performance of current trends over those run in the past. 

To Summarise

Automation testing has evolved into a key software delivery component and an integral part of the CI/CD pipeline. Apart from reducing feedback time, it provides real-time performance metrics, and helps maintain certified code versions that are ready to be shipped to customers as required. Automated test suites can help tests keep pace with ever-changing requirements, reduce cycle time, and increase adaptability to change. 

The overall high product sanity levels during release, fosters greater organisational confidence in their software and engenders a healthy development and release environment.

Run your CI/CD Tests on Real Devices