Most development and testing teams consider functional mobile UI tests as the last quality gate, executing only unit tests as a PR check just before merging code. As a result, these UI tests are usually only scheduled once a day or once per release.

But what if these tests were run more frequently, at each pull request? Would there be a significant improvement in release quality? Let's find out.

In this post, we will go over:

  • Why you should test at every pull request
  • How to choose what tests to run
  • How to set up a pipeline to automatically test and validate PRs

Why is running end-to-end UI tests for each PR important?

Running end-to-end UI tests for each PR can be extremely beneficial when done right.

At Carousell, it helps identify major issues caused by changes much earlier in the development cycle. After introducing this, we were able to identify some of the crashes and major UI bugs much earlier in the cycle. So, finding and fixing changes before a release becomes much easier.

Usually, an app's unit tests will mock external dependencies. When you run the E2E tests in a stable environment, it validates those integrations as well.

Lastly, running functional tests with unit tests bolsters developers' confidence in release quality.

Choosing the right tests for an efficient pipeline:

While there is no one-size-fits-all approach to choosing tests for an efficient pipeline, here are a few considerations:

  • Choose essential tests that don't take too long to run or increase developers' wait time before merging the code.
  • Cover the most critical workflows (P0 or basic) to ensure your application is not breaking essential functionality. Add other tests if you have the bandwidth.
  • Do not consider this a replacement for your unit test or regression suite. Instead, think of it as a subset of your sanity suite.

How we set up the pipeline at Carousell:

With these basics out of the way, let's look at how we set up the pipeline at Carousell.

The test workflow at Carousell
  1. We worked with the development team for a script to generate the mobile app build for each PR and pointed it to the most stable environment; production, in our case.
  2. We configured each script to run in a Jenkins job as a PR check. The build artifacts are uploaded to a Google Cloud Platform Bucket with a unique build version.
  3. We configured another chained job to run the end-to-end tests. The "Build Job" waits until the "test job" is completed before marking itself as complete.
  4. Based on the test results, the Github PR check is automatically marked as passed or failed.
  5. After the jobs finish executing, we notify developers about the results on Slack. We created a custom script that sends a message like the one below:

How to configure a Jenkins job to run for each Github PR:

Once the pipeline is in place, here's how you can configure the Jenkins job to run for each GitHub PR. We use a Jenkins plugin called GitHub Pull Request Builder. You can follow the installation steps on its readme page.

1. Once installed, create a Jenkins job. Our preferred method is a 'Pipeline Job.'

2. Configure the Github Project URL.

3. Under Build Trigger, select GitHub Pull Request Builder and configure it as shown below.

4. To make sure it displays the proper name in the PR check configure theCommit Status Context text, under Advanced > Trigger Setup.

5. Create a sample PR, and you should see a new PR check.

6. To mark it as a required check in GitHub, go to Settings > Branches > select edit of your main branch > select Require status checks to pass before merging as shown below.

And you're all set. We hope this helps you set up your own feedback cycle, put the right quality checks in place, and provide timely feedback to developers to improve the overall quality of the release.