How to implement a CI/CD Pipeline?

Let’s understand the basics of CI/CD, before optimizing the pipeline.

Continuous Integration (CI) is a software development methodology in which the code is integrated several times a day. Each code change is verified by an automated pipeline that offers early feedback to developers, in case of any bugs.

Continuous Delivery (CD) is the ability to introduce changes to code—bug fixes, new features, configurational changes and the like—into production, safely, quickly and sustainably. This can be achieved by ensuring that with every commit, the code is ready for production so that it can be deployed to production on demand and as a routine activity. This reduces the time of deployment during production. However, the risk of introducing bugs in the software increases with every code change. Hence, to achieve quick and secure deployment at production it is important to put safety checks or tests which can detect bugs before the code is promoted to production.

Continuously integrating and running automated tests on every code build gives a developer the assurance that the checked-in code will work as expected.

However, to ensure that the whole business functionality and integration between different services is successful, a QA Engineer has to write more automated test scenarios to test user journeys. By running these test pipelines with every code change, one can ensure that the journey is not broken at any point in time. If it is broken, the developer gets feedback on the same and knows which commit led to the anomaly.

While following a CI/CD pipeline, the main motive of a QA is to minimize bugs during production. This requires that certain software development practices be followed.

What practices does a CI/CD pipeline include?

• Test Pyramid

Building in a healthy mix of manual and automation testing ensures that a QA gets better coverage of all scenarios and receives early feedback regarding any failure. A test pyramid shows how tests are grouped as well as the number of tests to be written in each group.

Unit tests entail the testing of a small unit or functionally of the code.

Let’s take an example here –

Testing a method that returns the addition of two numbers is easier than testing the whole calculator. As we move up the test pyramid the number of test decreases and the time taken by tests increases. So, if there are fewer unit tests, the QA has to cover more regression test scenarios which will eventually increase the regression pipeline time. Consequently, the feedback cycle will also increase for developers. This is an anti-pattern named Ice Cream Cone Antipattern.

Source: Watirmelon Blog

In the Ice cream cone antipattern, a lot of effort is needed to write and manage regression tests. If any regression scenario is missed then it will directly fail in the production environment, so manual testing effort increases significantly. This eventually makes the QAs less confident about the code quality.

• Code Build with test

Every code commit done by the developer should have unit & integration tests done when new functionality is added to the code. With every code push, the code should be built with the automated test (Unit & Integration) in an automated build pipeline to ensure that the addition of new functionality is not breaking the existing build. It operates as a safety net as it does not allow promotion of new code if the unit/integration tests are failing on the CI tool. Code changes which can break existing functionality get easily detected by the tests.

• Environment for testing

The testing environment should be different for developers and QAs. The promotion of the code to the environment can be automated with a few test strategies or can be manually controlled by the QA. The development environment should not be used by the QA for testing.

What are the various tests in an automated test pipeline?

• Smoke Test

Smoke tests are preliminary tests that check if the critical functionality of the software is working as expected. In other words, the test replicates the user journey to validate if the website/app offers what was promised. Once smoke testing is completed, the code can be promoted to the next environment for regression testing. It provides early feedback as it is fast and goes through all the services of the website/app. Smoke testing can be automated by using an automation tool like Selenium.

• Regression Test

Regression Testing is used to confirm that a recent program or code change has not harmed the stability of existing features. Regression Testing is nothing but a full or partial selection of already executed test cases which are re-executed to ensure existing functionalities work fine. It is possible to test both manually and automatically.

Try Running Faster Regression Testing Suites

Manual regression tests require QAs to go through the entire process manually and check if the functionality is not broken in any manner. In automation, the same task is done by an automation tool like Selenium. It is better to write automation test scripts for this since running repeated regression tests manually might not be feasible unless one has an enormous QA team at their disposal.

• Cross Platform Testing

Cross Platform Testing is used to test if the software is compatible and works as expected on different platforms. For example, if the software is a desktop application that works on Windows 7, cross-platform testing will determine if it works on Mac and Windows 10 as expected.

• Cross Browser Testing

Cross Browser Testing involved testing a web application on different versions of multiple browsers to ensure that it is compatible with all of them. Essentially, one must ensure that the look, feel and functionality of the web app remains consistent across browsers. To do cross browser testing manually, one needs to have numerous devices at hand to simulate the behavior of the web applications on each of them. This may turn out to be a logistical nightmare. It is easier to automate the process with automation platforms such as BrowserStack.

• Performance Testing

Performance Testing evaluates the speed with which the software responds and even how much load it can take in terms of usage. Performance testing is a vital part of the software development process. Needless to say, slow-responding software will be quite undesirable to customers. Manually running performance tests is nothing short of unmanageable, so they are usually automated.

How should a developer set up their CI/CD pipeline for maximum output?

1. Run all unit and integration tests with every build in the pipeline. If a failure is encountered with any test, stop the promotion of the service to the deployment environment.
2. If all tests have passed and build of all the services has proved to be functional, an automatic trigger should be in place to deploy the services to the development environment. Additionally, smoke testing should be automatically initiated over the deployment. A smoke test will cover almost all the services and ensure that the core functionality is not broken due to changes deployed.
3. If the smoke tests are positive, promote the deployment automatically to the next environment i.e QA environment.
4. Run regression tests in the QA environment to test all other business scenarios which are not covered in the lower part of the test pyramid.
5. After this, also consider setting up a pipeline to run performance tests and cross-platform tests to ensure the software is performant and is working as expected in other systems.

Talk to a Solutions Expert

Did you know: Optimizely runs 15,000+ tests in 45 minutes

Optimizely scaled its automation testing by moving from an on-premise grid to our cloud infrastructure.

Talk to a Solutions Expert

What practices should one avoid?

• Duplication

Duplication of test scenarios over multiple test suites leads to slowness of the regression/automation test pipeline. If one single scenario is tested simultaneously across multiple tests which will run for hours, eventually the feedback cycle will increase drastically. After the test time crosses a certain limit no one will wait for feedback and the running that test suite will lose all meaning.

• Limited automation

Automating anything and everything including test scenarios that have been covered in the lower levels of the test pyramid can cause serious trouble when it comes to maintenance. Since even a minuscule change in a feature can lead to the failure of many regression test suites, it will, in turn, lead to increased efforts to fix it i.e increased overhead of automated tests.

• Only QAs owning test pipelines

It is hard for only QAs in a team to own all the QA test pipelines. With constant change in the software, it is very likely that the automated scenario will start failing after a particular commit, so it’s a good practice for developers to also monitor automation pipelines and fix bugs when they can.

• Unaddressed test pipelines

The purpose of writing automated tests is to get feedback as early as possible in the development cycle and to test a vast number of user scenarios before going into production. If the tests are failing and no one in the team is resolving them then there is no benefit of creating an automated test pipeline. In this case, the whole point of automating the user journey and testing software functionality will go in vain.

Talk to an Expert

Final Thoughts

To get the most out of a CI/CD strategy, consider using testing platforms like BrowserStack Automate which seamlessly integrate into developers’ CI/CD pipeline to help agile teams scale up.

Implementing a CI/CD pipeline is non-negotiable at this point. In order to implement discipline, efficiency and high levels of quality control in software development, CI/CD provides tools and workflows that work in favor of developers. A discerning combination of manual and automated tests can structure a CI/CD pipeline so as to streamline development cycles and provide superior user experiences.