How to run parallel tests with CircleCI

More often than not, testers have to deal with a large codebase comprising hundreds or even thousands of tests. Most tests validating website or app functionality must be rerun on different platforms (devices, browsers, operating systems). 

However, running so many tests on different platforms can be immensely difficult, not to mention tedious and error-prone, without the right strategy and tech. The best and easiest way to test under such conditions is to use parallel testing. 

This article will explore how to run parallel tests with a particularly popular and useful CI/CD tool like CircleCI. 

But, first thing’s first.

What is parallel testing? 

In parallel testing, you test different modules or applications on multiple browser-device-OS combinations simultaneously rather than one after another. In a scenario where we have two versions of software available, and both must be examined to ensure stability and compatibility, it is easier to run tests on two versions simultaneously, get results and detect bugs faster

Parallel testing reduces execution time and effort, resulting in faster time to delivery. It is particularly useful in the case of cross browser testing, compatibility testing, localization, and internalization testing. 

How to run tests in parallel with CircleCI

Before exploring parallel testing with CircleCI, it’s important to place a quick note on why it’s important to run parallel tests on real browsers and devices. 

While it may seem easier to download and test on emulators and simulators, they are riddled with inadequacies that prevent them from providing a reliable test environment. For example, they cannot mimic low network conditions, low battery, or incoming calls. In general, they cannot replicate all features of real browsers and devices, making the results of tests run on them prone to serious error. 

Simply sign up for free, log in, select the browser-device-OS combinations to be tested on, feed the relevant tests scripts and let the tests run themselves via automation. 

Test on Real Browsers & Devices

Now, on to CircleCI. 

CircleCI provides us with capabilities to run tests in parallel and achieve faster completion.

When you run your tests with CircleCI, they usually run on a single VM. Now, the more tests, the more time it will take to complete on one machine. In order to reduce this time, you can run tests in parallel by distributing them across multiple separate executors. 

In order to do this, you need to specify the parallelism level to define how many separate executors should be spun up for your job. After that, for separating the test files, you can either use CircleCI CLI or environment variables to configure each machine individually.

How to set up parallelism in CircleCI

Test suites are usually defined at the job level in your .circleci/config.yml file. So let’s first set up the parallelism level on your config file. 

This parallelism key defines how many independent executors CircleCI will create to run your tests. To do this, you just need to add this key before calling the steps to run your tests. 

Depending on your CircleCI’s pricing plan, the parallelism value can be increased to match your requirements.

To run a job’s steps in parallel, set the parallelism key to a value greater than 1.

# ~/.circleci/config.yml

version: 2

jobs:

  test:

    docker:

      - image: cimg/<language>:<version TAG>

        auth:

          username: yourdockerhub-user

          password: $DOCKERHUB_PASSWORD  

    parallelism: 4

Now, in order to split the actual tests, the next step is to use the CircleCI CLI (Command Line Interface) to call additional commands to split the tests.

Run Parallel Tests on Real Devices

How to split test files using CircleCI CLI

CircleCI supports automatic test allocation across your containers. The allocation is based on filename or class name, depending on the requirements of the test-runner we are using. It requires the CircleCI CLI, which is automatically injected into your build at run-time.

The commands that we will use for splitting the test files are glob and split. 

• A glob is basically a pattern that will match a list of filenames to a specific pattern that you specify.
• The split command, by default, will split your tests according to a list of filenames, but you can also use other strategies such as splitting it by timing data or file size.

By using these two commands, you can split your tests into multiple executors that are independent of each other as you can see from the image above.

Globbing the tests

In order to assist in defining your test suite, the CircleCI CLI supports globbing test files using the following patterns:

• * matches any sequence of characters (excluding path separators)
• ** matches any sequence of characters (including path separators)
• ? matches any single character (excluding path separators)
• [abc] matches any character (excluding path separators) against characters in brackets
• {foo, bar,…} matches a sequence of characters if any of the alternatives in braces matches

So to glob test files, pass one or more patterns to the CircleCI tests glob command.

Now, let’s look at how we can achieve this in our configuration file.

circleci tests glob "tests/unit/*.java" "tests/functional/*.java"

You can use the glob command in your config file with the echo command as shown below, to check the results of pattern match:

# ~/.circleci/config.yml

version: 2

jobs:

  test:

    docker:

      - image: cimg/<language>:<version TAG>

        auth:

          username: mydockerhub-user

          password: $DOCKERHUB_PASSWORD  # context / project UI env-var reference

    parallelism: 4

    steps:

      - run:

          command: |

            echo $(circleci tests glob "foo/**/*" "bar/**/*")

            circleci tests glob "foo/**/*" "bar/**/*" | xargs -n 1 echo

Splitting the tests

The split command, by default, will split your tests according to a list of filenames but you can also use other strategies such as splitting it by timing data or file size. Let’s discuss all of the splitting techniques.

Split By Name

This is the default splitting technique. So in case you don’t specify a method using the –split-by flag, CircleCI expects a list of filenames/classnames and splits them alphabetically by test name. 

There are a few ways to provide this list:

• Create a text file with test filenames

circleci tests split test_filenames.txt

• Provide a path to the test files

circleci tests split < /path/to/items/to/split

• Pipe a glob of test files

circleci tests glob "test/**/*.java" | circleci tests split

Once the list of names has been provided, CLI looks up the number of available containers, along with the current container index, and then it uses the deterministic splitting algorithms to split the test files across all available containers.

Split By Time

This is the best way to optimize your test suite across a set of parallel executors. This ensures that the tests are split in the evenest way, leading to overall reduced test time.

In order to split by test timings, use the –split-by flag with the timings split type. The available timings data will then be analyzed and your tests will be split across the parallel-running containers as evenly as possible.

circleci tests glob "**/*.go" | circleci tests split --split-by=timings

On each successful run of a test suite, the timings data is saved from the directory specified by the path in the store_test_results step. This timings data mentions how long each test took to complete per filename or classname, depending on the language used.

One thing to be considered is if you do not use store_test_results, there will be no timing data available for splitting your tests.

The CLI expects both filenames and classnames to be present in the timing data produced by the test suite. By default, splitting defaults to the filename, but you can specify classnames by using the –timings-type flag.

cat my_java_test_classnames | circleci tests split --split-by=timings --timings-type=classname

For partially found test results, we automatically assign a random small value to any test we did not find timing data for. You can override this assigned value to a specific value with the –time-default flag.

circleci tests glob "**/*.rb" | circleci tests split --split-by=timings --time-default=10s

Additionally, If you need to manually store and retrieve timing data, use the store_artifacts step.

Splitting by filesize

You can also split tests by filesize. To do this, use the –split-by flag with the filesize split type.

How to split tests using environment variables

CircleCI provides us with two environment variables that can be used in place of the CLI to configure each container individually. 

They are CIRCLE_NODE_TOTAL and CIRCLE_NODE_INDEX.

• CIRCLE_NODE_TOTAL is the total number of parallel containers being used to run your job. 
• CIRCLE_NODE_INDEX is the index of the specific container that is currently running. The environment variables ensure us full control over parallelism.

How to run split tests

Now after globbing and splitting the tests, it’s time to run them. To combine test grouping with test execution, you can consider saving the grouped tests to a file and then passing this file to your test runner.

circleci tests glob "test/**/*.rb" | circleci tests split > /tmp/tests-to-run

bundle exec rspec $(cat /tmp/tests-to-run)

The contents of the file /tmp/tests-to-run will be different in each container, based on $CIRCLE_NODE_INDEX and $CIRCLE_NODE_TOTAL.

Applying test parallelism using CircleCI regardless of your underlying technical approach is always beneficial because it will speed up your overall execution time and in some cases, can even reduce the time by half. 

Run Parallel Tests on BrowserStack for Free

Utilize the steps in this article to run parallel tests in CircleCI, thus getting faster results and integrating speed into your CI/CD pipeline with ease.