Build status calculation and visibility of tests and hooks

Learn how build status is calculated in Test Observability.

Build status

Build status in Test Observability helps you quickly identify how your builds and the associated tests ran.

Test Observability marks every build into one of the following statuses based on the status of the tests in the build:

• Passed
• Failed
• Skipped
• Running
• Unknown

When a build starts, Test Observability receives a build start event and temporarily marks the build status of the build as running. When the build runs, it receives multiple events from the testing framework one after the other. Test Observability continues to listen to all these events until it receives the build finish event or when an Inactivity timeout occurs.

The following table details how Test Observability determines the status of a build:

Inactivity timeout

When you run a build, Test Observability receives multiple events from your test executions.

If the waiting time between receiving two events breaches a predefined limit, the build times out due to inactivity. This could happen due to issues like intermittent network connectivity issues, abrupt CI job crashes, and several other reasons. When inactivity timeout occurs, if Test Observability has not received the test finish event of any test, it marks the status of such tests as unknown.

Modify the inactivity timeout limit

By default, the inactivity timeout limit is 300 seconds. You can change this predefined limit as per your requirements.

Follow these steps to modify the inactivity timeout limit in Test Observability:

1. Go to Settings > General.

2. Update Timeout for build and click Save changes.

You can contact support if you continue to see that tests are marked as unknown even after modifying the inactivity timeout limit.

Visibility of hooks and tests

Test count in Test Observability

Test Observability maintains the total number of tests in a build and also the count of passed, failed, skipped, and unknown tests.

You can view the total test count in the Build Summary widget in Build Insights. You can also view the number of passed, failed, skipped, and unknown tests at the top-right-hand corner of the Build Insights page and also in a few other widgets.

The following screenshot highlights the test count and the test status count displayed in Build Insights:

How hooks are shown in Test Observability

Test Observability handles hooks that run with each test case (like beforeEach and afterEach) differently from the hooks that run once for a block of test cases or at a file level (global hooks like beforeAll, and afterAll).

Hooks that run with each test case are always excluded from the test listing by Test Observability. You can view such hooks only inside a test.

In contrast, Test Observability provides you with two options to handle the hooks that run once for a block of tests or at a file level (global hooks):

• Show only failed hooks in the test listing so that you get visibility on all types of failures.
• Show all hooks (both passing and failing) in the test listing for complete visibility.

How to include all hooks in the test count and metrics

For hooks that run once for a block of tests or at a file level (global hooks), Test Observability shows only failed hooks in the test listing by default. This is the recommended setting.

However, you can include all such global hooks in the test listing. This setting will alter the behavior of some features and the way Test Observability calculates certain metrics.

Following are the main features and metrics that will change when all global hooks are included in the test listing:

• Test Listing will show such passed global hooks as separate tests.
• Aggregate test count and the widgets in Build Insights will include such passed global hooks as separate tests.
• Tests Health metrics like Failure count and Failure rate will include passed global hooks as separate tests.
• Testing Trends metrics and widgets will include passed global hooks as separate tests.

You can follow these steps to treat all hooks as tests in Test Observability:

1. Go to Settings > General.

2. Select Include all hooks under Hooks visibility.

3. Click Save changes.

Handling hooks within custom alerts

Test Observability allows users to set up custom alerts for every build. You have the option to include all hooks, or only failed hooks in the calculation of some of these alerts. The way these alerts are triggered depends on whether you include only the failed hooks or both passed and failed hooks in the calculation.

The table below clarifies the behavior of various alerts based on including all hooks or only failed hooks in the calculation: