To help engineering teams verify their builds faster, BrowserStack is pleased to announce the launch of the Quality Gate feature in Test Observability! 

All engineering teams want to move faster, but they often lack confidence when promoting code from one environment to another or merging code. Confidence that features won't break and that the pipeline won’t be slowed by non-issues.

Once code is committed or a PR is created, every team scrambles to get automated tests passing so they can move the code to production. Over the years, BrowserStack’s Automate & App Automate products have helped users drastically reduce build time by enabling test parallelization at scale. However, when tests often fail with false negatives, it increases the time teams spend verifying builds before sending a PR on its way.

Quality Gate is designed to completely eliminate manual verification of builds, enabling PRs to be merged or deployed to production automatically. Users can configure a number of quality rules that run once a build is complete and automatically block or allow code merges and deployments.

You can get started with Quality Gate & Test Observability here!

Also check out the Quality Gate documentation to learn more!

Here's what one of our early users of Quality Gate had to say about the feature:

Using BrowserStack Quality Gate, you don't need to rely solely on the 'test case run status' anymore but can target something more fundamental - which is 'errors'. This is a much robust feedback cycle than before.

This, in my humble opinion, is a game changer!

- Pramod Yadav, Test Lead @ Mercell

How Quality Gate enables continuous deployment and faster verification

Test Observability is uniquely positioned to help users precisely instrument and surface insights for all types of tests - Functional, Unit, API, or others — whether the test runs on BrowserStack infrastructure, a developer’s laptop, or an in-house browser grid. 

With an intuitive rule builder and a ready-to-integrate API, you can set up Quality Gate on Test Observability in minutes! If you’re not set up on Test Observability yet, we support a wide range of test types and test frameworks (from Playwright to JUnit XML Reports) and require zero code changes to your existing setup to take you from CI to CD.

Here's a quick overview of how Quality Gate fits into your pipeline:

With Quality Gate, teams can configure rules based on their specific priorities to ensure high standards of quality. Here are some examples of automation you can put in place:

  • Allow a deployment if all your P0/P1 test cases pass but a few P2/P3 test cases fail.
  • Implement an instant roll-back tool if tests are breaking in Production.
  • Enforce organization-wide best practices on flakiness, test performance, and more.
  • Set thresholds for different failure categories, such as Product Bugs, Automation Issues, Network Issues, etc. — all powered by our ML-based auto failure analysis.
  • Ensure a minimum number of tests run (especially your critical tests) before allowing code merges.

Building Quality Gate 

Understanding the need for Quality Gate

While building Test Observability and speaking to countless customers, we realized that the tradeoff between confidence and pace is a common dilemma; one without an apparent solution. We spoke to multiple customers who each phrased the same issue in different ways without realizing that a Quality Gate is the solution. 

Customers would come to us with a simple question, “Do you happen to have an API?” prompting us to ask, “Yes, but what would you like to do with it?”

Here are some of the responses:

  • "I want to give developers visibility in PR checks on where failures came from."
  • "We need to ensure we don’t let too many flaky tests into a merge or deploy."
  • "I have a bunch of always failing tests I don’t care about. How can I stop my pipeline from failing for these?"
  • "There must be multiple ways to gauge a build other than just pass/fail tests. We would like to access a few different metrics to let developers know if a build is good to go or not."

When we asked users what was stopping them from going full CI/CD with this information, we realized the gap that we needed to fill. We realized that giving our users all the information they want - plus the ability to set up automation in a single API or plugin they could add to their pipeline was what they needed. So we that's what we did!

The Capabilities of the Quality Gate 

Configuring Rules & Profiles

We wanted users to access as many fine-grained metrics as possible while building CI/CD automation with Quality Gate.

Test Observability offers the following rules out of the box, with filters available for each rule, to target individual tests or classes either statically or dynamically.Users can group multiple rules into Quality Profiles to target builds dynamically, such as:

    • Unique Errors - Check for regressions in terms of errors or stack traces being thrown across builds.
    • Test Status - Set absolute or relative thresholds on the various test statuses in your build.
    • Smart Tags - Set absolute or relative thresholds for metrics like Flakiness, New Failures, and more.
    • Failure Category - Set thresholds for ML-analyzed failure categories.
    • Alerts - Check for the presence of pre-configured alerts.

The BrowserStack Default Quality Profile

To help users get started faster, the BrowserStack Default Quality Profile comes with 4 rules out of the box.

  1. No tests are marked with the New Failures Smart Tag
    This ensures that no new tests start failing in a build.
  2. No New Unique Errors are detected from the last build
    This ensure that there is no regression of errors from the last build creeping in (this is to ensure that even if no new tests failed, no new error crept in).
  3. No tests are automatically marked with the Product Bug failure category
    This ensures that no tests that are ML-analyzed to be Product Bugs are allowed to creep into a merge or deploy.
  4. Tests marked with the Flaky Smart Tag is less than 15%
    Flakiness is part and parcel of every build but has to be limited. Based on millions of tests run on Test Observability, we’ve found that this is a good threshold for users to start with.

Based on extensive user research and millions of data points from Test Observability tests, we believe that these represent a well-rounded set of best practices for users to build into their Quality Profiles. We hope it can serve as inspiration for users to build more nuanced Quality Profiles that can help them take their automation testing to the next level.

Integrating into your CI/CD workflow

You can integrate the Quality Gate status into your CI/CD or SCM tool with a simple API call to Quality Gate. You can get started in minutes with sample recipes for your pipeline, which will automatically fetch the Quality Gate result and pass or fail your CI/CD job.

What next?

We will continue to innovate in this area and provide new ways for you to use the Quality Gate in your existing tooling or via new rules! Try out Quality Gates with Test Observability today and share your feedback!