Accelerate CI/CD pipelines with Parallel Testing

Continuous Integration / Continuous Delivery (CI/CD) processes came into existence to help solve a myriad of problems in the SDLC. Among them, one of the major pain points for organizations is how to speed up the development and release cycles and maintain the quality of software simultaneously. CI/CD helped solve it by accelerating the release cycle through Automation of Build and Release processes while implementing test automation. This automation of Build, Release, and Testing formed the foundation of CI/CD pipelines.

Ever since the implementation of CI/CD has begun, organizations have been exploring ways of making their pipelines faster. Here are some of the strategies that have been widely implemented:

• Build what you need: breaking down a large build into small chunks.
• Test minimalistic: test only the basic features on every commit.
• Parallel testing: Perform tests parallelly when running an extensive test suite on two different versions of the application.
• Deployment Automation: Automate all deployments through configuration management tools and perform deployments on multiple servers at once.

This article explores the aspects of Parallel Testing and the various options that can be leveraged to perform parallel testing.

Parallel Testing in CI/CD

In the early days of testing, QAs manually performed tests on every step of the application and assert the behavior. Manual Testing is cumbersome and it also takes a considerable amount of time. Moreover, it is prone to human errors.

Hence, all the shortcomings of Manual Testing were addressed by test automation. With the increasing adoption of Test Automation suites, more frameworks were created to accelerate testing.

However, due to limited test infrastructure, testing teams had to run all these tests sequentially. Sequential running of tests became a time-consuming process, especially when the test case volume was high and device coverage was wider. It led organizations to delay releases or even go live without enough testing of the software. One of the most viable solutions to this issue was to Run independent pieces of test parallelly.

Parallel testing in the CI/CD context is either running multiple tests parallelly, performing cross-browser testing simultaneously on multiple browsers, or a combination of both. Running cross browser tests in parallel using Selenium Grid, enables the user to test across multiple browser-device combinations. This empowers the QA to increase their test coverage without compromising the delivery timelines.

Benefits of Parallel Testing in CI/CD

Some of the core benefits of Parallel Testing include:

• Faster testing: Parallel testing helps run multiple independent test cases parallelly. Also, long-running test cases take up a considerable amount of time when run sequentially. Such test cases can be triggered parallelly, thus reducing the total time to complete the run.
• Better use of infrastructure: Allocation of high-performance hardware and running tests sequentially does justice to the capability of the hardware very seldom. Also, running high-performance servers for a longer amount of time to run tests sequentially increases the cost substantially. Instead, when tests are configured to run parallelly, the underlying hardware runs to its capacity, and since the tests complete running faster, the servers can be stopped, and costs can be reduced.
• Wider Coverage: Organizations often refrain from running a lot of tests owing to the time constraint. When tests are run parallelly, the total time for tests reduces. This allows for performing a larger number of test cases, thus providing more test coverage for the applications.
• Accelerates the CI/CD process: Testing takes a major chunk of time in almost all the CI/CD pipelines. When testing happens parallelly, it reduces the time and thus also reduces the total time taken for the pipeline to run.

Parallel Testing Tools and Frameworks

Multiple testing tools/platforms offer modules for running parallel tests. Selenium Grid and Browserstack are well-known platforms.

Selenium Grid

Selenium Grid allows running test cases in different machines across different platforms simultaneously. The triggering of these test cases is controlled on the local end. The test cases upon trigger are executed automatically by the remote end.

With the advent of WebDriver tests, one might need to run the tests on multiple browsers and operating system combinations to check cross browser compatibility in a shorter span of time. Selenium Grid support parallel testing across different browsers.

BrowserStack

BrowserStack has a real device cloud that supports cross-browser testing. It enables developers to test their websites across various browser versions on different desktop and mobile devices. Users can use BrowserStack Automate test on 3000+ browser-device combinations without the need to install virtual machines, devices, or emulators. This makes it a one-stop platform to test web apps on multiple browser versions and operating systems parallelly and seamlessly.

Try Parallel Testing for free

Leverage Parallel Testing Infrastructure

Often, the infrastructure allocated to run automated tests becomes a limitation to the speed and depth of testing. At the same time, even some of the well-to-do companies spend a lot on test infrastructure, which will be barely used to its optimum capacity. Striking a balance between these two is a challenging task. This is where maintaining in-house infrastructure becomes a huge overhead with limited ROI.

DevOps engineers need to work with QA teams to understand the frequency and intensity of the tests. The infrastructure required to perform parallel tests can be accordingly planned and tests can be scheduled in a way to make the best use of the underlying hardware. Here are a couple of strategies that can be put into place to accelerate the test execution.

Check Impact with Parallel Test Calculator

Parallel Execution of Test Cases

Parallelization of test cases can be done to help complete tests faster. Tests that are independent of each other can be executed parallelly so that they can use the infrastructure available to the maximum capacity and complete the executions faster.

Parallel Execution of Test Suites on Independent Infrastructure

For long-running tests that need to be executed on multiple versions of the code, separate test environments can be created on the fly. This can be done using containers. The ephemeral nature of containers can be put to use to create a temporary test environment to run a test suite, and once done, the containers can be brought down. This can be set up on bare metal servers using Docker, or cloud services like Amazon ECS, Google Container Engine(GKE), Azure Container Service, etc., can be used.

The Way Ahead

After discussing multiple options to run parallel tests in CI-CD, it is important to diligently apply one or a combination of the above as it fits the requirements of the project. To choose the optimum solution for parallel testing in CI/CD, the DevOps team and QA team should collaborate together and analyze various aspects. This will help the tests complete faster, thereby, accelerating the whole CI/CD pipeline.

• Post every commit, a set of unit tests can be run, and a basic functionality test suite can be executed against the new build artifact. At this phase, using the Parallel execution of test suites on an independent infrastructure method is recommended.
• When multiple versions need to be compared against each other for performance assertion, using parallel test environments that spun together using containers is a viable solution. Although, data management for these parallel test environments is a challenge and needs to be chalked out carefully.
• Running regression tests at the end of every day is a best practice. Regression tests could be vast and might contain a large number of scenarios. In this case, Parallel execution of test cases methodologies is suggested. Independent test cases can be scheduled to run parallelly using Selenium Grid.
• While performing cross-browser testing, using a combination of both parallel test execution and parallel infrastructure is a best practice to improve efficiency. Containers can be created with different WebDrivers to support multiple browser testing, and these can run parallel.

Owing to the need for accessing multiple platforms and browsers to run tests in parallel, the cost of compatibility testing with parallel testing increases, when going for an in-house testing infrastructure. Also, one might come to a point where access to all browsers and versions may not be possible and coverage becomes limited in such cases.

However, this problem can be solved using a secured public real device cloud like BrowserStack provides access to numerous platforms and browsers with their corresponding versions on the cloud. BrowserStack Automate enables QA to run automated parallel tests and use multiple browsers and versions. Testing on real devices can help in identifying bottlenecks by testing under real user conditions, improving the overall testing accuracy. This way, one can test on different browser-device combinations in parallel, reducing the time taken to run tests. Thereby, accelerating the CI/CD pipelines.

Run Parallel Tests on Real Device Cloud for Free