Quality Profiles & Rules Explained

Learn how to configure and use Quality Profiles & Rules

Quality Profiles

Quality Profiles are collections of rules you can create, which will run as part of a Quality Gate analysis when your build finishes. Use Quality Profiles to group relevant rules that help you target specific builds, or even specific tests within builds.

Targeting Builds and Projects with Quality Profiles

• You can choose to have your Quality Profile apply to All Projects or just the Current Project you are configuring that Profile within. This will help you enforce quality standards across your organizations, or limit the scope of a Quality Profile to only a project where it is relevant. Please exercise caution while editing Quality Profiles that target All Projects, as this could affect other teams that use the Quality Gate feature.

• You can also choose to have your Quality Profile apply to All builds within a project, or target specific builds either by Build Name or Build Tags. This will help you isolate your Quality Profiles to wherever they are relevant.

Passes If or Fails If Quality Profiles

• Select the type of Quality Profile before configuring rules: Passes If or Fails If.
• A Passes If profile requires all rules to be true to pass. If any rule is false, the profile fails. (E.g. A profile contains 5 rules, 1 rule is evaluated to false, then the profile fails). This type of profile can be used to ensure that all rules meet the baseline thresholds or requirements for a build.
• A Fails If profile fails if a single rule is true. (E.g. A profile contains 5 rules, 1 is evaluated to true, then the profile fails). This type of profile can be used to look for breaches in your build that are unacceptable for you.
• Both profiles can achieve the same outcome, and allow you to construct them in a way that you would like to evaluate a build in your organization.

Rules

Test Observability offers a number of rule definitions which you can use to craft precise a Quality Gate that will help you ensure your builds meet organizational standards.

Filters

Almost all rules come with the option to apply filters to have them apply to Any tests or Selected Tests. This helps you create rules that target tests of significance (e.g. Ensuring that no P0 or P1 tests failed, while ignoring P2 or P3 tests.)

You can currently filter tests by Folder or by Tag.

Rule Types

Test Status

• This rule helps you monitor the presence of certain test statuses in your build.
• You could use this rule to ensure certain tests passed, or ensure certain tests didn’t fail or have an error.

Test Status Threshold

• Set absolute or relative thresholds on the various test statuses in your build.
• You could use this to set a minimum passing or failing threshold for the whole build or at a class or folder level as well.

Test Executions Threshold

• Check how many tests executed in your builds.
• You can use this rule to to ensure all your expected tests have run, or critical tests have run.

Stability

• Set thresholds for the stability of your build or specific tests.
• Stability is defined as the percentage of passing tests.

Smart Tags

• This rule helps you monitor the presence of certain smart tags mapped to tests in your build.
• You could use this rule to ensure that tests didn’t get mapped with tags like “Flaky”.

Smart Tags Threshold

• Set absolute or relative thresholds on the various smart tags mapped to tests in your build.
• You could use this to set a minimum threshold for smart tags for the whole build or at a class or folder level as well.

Alerts

• Make your alerts actionable! Leverage your already configured alerts to fail the Quality Gate.

Automatic Failure Analysis - Failure Categories Present

• This rule helps you monitor the presence of failure categories (ML-analyzed upon build completion) in your build.
• You could use this rule to ensure that tests didn’t get mapped with failure categories like Product Bug.

Automatic Failure Analysis - Failure Categories Threshold

• Set absolute or relative thresholds on the various failure categories (ML-analyzed upon build completion) mapped to tests in your build.
• You could use this to set a minimum threshold for failure categories like Product Bug or any custom category for the whole build or at a class or folder level as well.

New Unique Errors

• Discover if new errors have creeped into your build from the last N runs.
• You can use this as a way to prevent ‘error regressions’.

Unique Errors - Threshold

• Set thresholds for the number of unique errors in a build.
• This can help you set a cap on how many errors you are willing to accept in a build.