Automatic Failure Analysis

Learn how Automatic Failure Analysis works and how it helps to speed up debugging and root cause analysis.

The Automatic Failure Analysis feature intelligently categorizes test failures, allowing you to quickly differentiate genuine failures from noise.

Prerequisites

Automatic Failure Analysis is triggered after the completion of each build run. But it will work only if:

1. There are failures in the current build run.
2. There should have been test failures in the same project in the past.
3. You should have manually tagged test failures into one of the failure categories in the past.

How to tag failures into categories manually?

Initially, all failed tests would be tagged as To be investigated. You can select one of the following categories from the list next to the test name:

• Product bug
• Automation bug
• Environment issue
• No defect

You can also create custom failure categories and select those as well.

Bulk tag multiple tests

You can tag multiple tests into a failure category using the following steps:

1. Click the menu next to a test.
2. Click Bulk update failure category.
   
3. Select the failure category from Failure category options.
   
4. Click Current Build, Last 10 Builds, or Last 10 Days to select the tests from a range.
   
5. Select the tests you want to tag the failure category and click Apply.
   

You can see a success message Tests updated successfully! to confirm that the tests are tagged to the failure category.

How does it work?

Automatic error analysis for failure category detection works by employing a host of sophisticated algorithms. It analyses past errors and considers the category that you might have assigned manually to the different errors.

Automatic error analysis learns with every instance where you tag errors manually and over time it becomes more and more adept at intelligently categorizing issues. To add to that, you can also create custom sub-categories through Settings and assign different failures to those.

How does it help?

As an automation engineer responsible for the quality of features that go into production, the single most important task is to ensure that no real product bugs creep into production and impact the end user’s experience.

In the process to ensure quality, automation engineers across the globe write tons of tests to verify that scenarios work fine. But, the biggest problem faced is that of flakiness and other non-product-code related issues which result in test failures.

Hence, it becomes super important for you to know which test failures are genuine and which are just noise. We have built the Automatic Failure Analysis feature keeping exactly that in mind.

Automatic Failure Analysis helps by doing the following:

1. Categorizes issues into Product bugs, Automation Issues, and Environment issues for you to identify which ones you can ignore from your analysis.
2. You can pinpoint test failures due to environment issues and it helps you decide the next course of action like re-running those tests.
3. You can also get a list of all Automation issues and either Mute them or create an issue on a project management tool such as Jira.
4. You can track long-term trends in the failure categories using Testing Trends. Using the insights from long-term trends, you can analyze what kind of bugs are creeping in, and work with the right teams to improve development and testing approaches.

Configure parameters of automatic error analysis

You can configure settings for automatic error analysis and control how it works.

How to create custom failure categories?

You can create custom failure categories for each project by navigating to Settings > Failure Categories.

Here, you can edit existing categories if you have any, or add a new category:

• Add a new category by clicking the + Create sub-category button.
• Edit or Delete an existing sub-category by clicking the kebab menu for each category.

Configuring custom failure categories is easy:

1. Select the failure category from the dropdown menu under which you want to create a custom failure category.
2. Set a name for your custom category.
3. Click Create or Save Changes to finish setting up your category.