What are Smart Tags?

BrowserStack Test Observability automatically tags your tests with Smart Tags to help you identify issues in your test suite faster. You can customize Smart Tags with custom rules so you can adapt them to suit your needs.

Test Observability currently supports the following Smart Tags:

• Flaky
• New Failure
• Always Failing
• Performance Anomaly

All Smart Tags are enabled by default and come with default configurations that should suit most teams. However, you can turn off Smart Tags selectively and customize Smart Tag configurations via the Settings menu.

Smart Tags are applied at the test case level. However, they are also shown at the build level for builds with test cases that have Smart Tags applied.

Flaky

Test Observability helps you identify tests that are flaky, and automatically Smart Tags them as “Flaky”. Any builds with flaky tests that meet the flakiness rules will also be tagged.

Conditions for the Flaky Smart Tag to be applied

Tests will be marked Flaky if either of the two conditions below are met:

1. By default, if the test passes on a retry attempt in the same run across the last 10 consecutive runs. You may customize this and set a value anywhere between 0 to 30. Setting this to 0 means a test would be marked as flaky only if it passes a retry in the latest run.
2. By default, if the test status flips from pass to fail (or vice versa) more than 50% in the last 10 runs. You can customize both the % value and the last X runs parameters. The maximum value you can set for X is 30.

New Failure

Test Observability automatically identifies brand new test failures previously unseen or recently unseen so you can quickly understand if something new has broken in your application.

Conditions for the New Failure Smart Tag to be applied

Tests will be marked New Failure if the following condition is met:

• By default, any test that fails with a new error for the first time in the last 5 runs.
• However, you can reconfigure this Smart Tag to be applied for any test failing with any error or new error in the last X runs.

Always Failing

Test Observability automatically identifies test cases that have been continuously failing and need repair.

Conditions for the Always Failing Smart Tag to be applied

Tests will be marked Always Failing if the following condition is met:

• By default, any test failing with the same error in the last 5 runs.
• However, you can reconfigure this Smart Tag to be applied for any test failing with any error or the same error in the last X runs. You can set the value of X between 2 and 30.

Performance Anomaly

Test Observability automatically identifies test runs that took much longer to execute and could be contributing to performance issues in your suite.

Conditions for the Performance Anomaly Smart Tag to be applied

Tests will be marked Performance Anomaly if the following condition is met:

• By default, any test with a run duration above the 95th percentile among the last 100 executions.
• However, you can reconfigure this Smart Tag and customize both the percentile value (between 80th and 99th percentile) and the number of executions (between 30 and 100 executions).

Test Observability takes into account a maximum of 100 days of data for this calculation.

Customizing Smart Tag settings

You can turn off Smart Tags selectively and customize Smart Tag configurations via the Settings menu. Configurations are maintained at a project level. For each unique project you have on Test Observability, you can configure and maintain separate settings.