JUnit XML or BDD-JSON based report upload

JUnit XML reports have set a benchmark in test result reporting and are widely adopted across various testing frameworks. This standardization extends to several programming languages, including Java, JavaScript, Ruby, and Python, all capable of producing JUnit XML formatted reports. Similarly, BDD-JSON reports are an important standard for Cucumber framework reports as they provide a detailed and easily interpretable format for test results.

You can create test reports using multiple frameworks through CLI and upload them to the Test Management from your terminal.

Understanding Test Run data handling

Test Run data selection using JUnit XML reports

A JUnit XML report typically includes details of automated test cases that were run as part of your test scripts. This includes Test Case titles, Test Case folder, Test Case execution time, Status (Pass, Fail or Skipped), and error responses in case of failures. The Test Runs that JUnit XML reports represent aggregate individual Test Cases and their results. The report details the status, test steps, and results.

Data import mechanism

• Test Case Mapping:

  Test Case titles, Test Case folder, Test Case execution time, Status (Pass, Fail or Skipped), and error responses in case of failures for each of the Test Cases that are imported in Test Management.

• Creation or association:

  A Test Case is either mapped to an existing Test Case (by ID or Title) or a new Test Case is created.

JUnit XML to Test Management mapping

JUnit XML report tags is mapped to Test Management as follows.

• The test result status is defined by the child tags of the <testcase> tag. By default, the status is Passed. If there is any of the failure, error, or system-err tag, the test result will be Failed, and if there is a <skipped> tag, the test result will be Skipped.
• The time at the folder level is cumulative of each Test Case execution time.

Examples for a failed test, error response, and skipped test

Failed test example:

In this example, the test suite MyTestSuite contains two test cases. The first test case, testMethod1, fails because the expected value of 10 does not match the actual value of 5. The second test case, testMethod2, has passed. Therefore, in test management, MyTestSuite is mapped to a folder, testMethod1 test case status is set as Failed, and testMethod2 test case status is set as Passed.

Error response example:

In this example, the test suite MyTestSuite contains one test case, testMethod. The test case fails with an error because it could not read the file /tmp/myfile.txt. Therefore, in test management, MyTestSuite is mapped to a folder, and the testMethod test case’s error message is copied to the Response field of the test case.

Skipped test example:

In this example, the test suite MyTestSuite contains two test cases. The first test case, testMethod1, passes. The second test case, testMethod2, is skipped because it is not relevant to the current environment. Therefore, in test management, MyTestSuite is mapped to a folder, testMethod1 test case status is set as Passed, and testMethod2 test case status is set as Skipped.

Import test case fields from JUnit XML reports

You can import specific test case fields directly from JUnit XML reports. These fields, such as State, Priority, Tags, Configurations, Type of Test Case and custom fields provides enhanced traceability, better categorization, and richer context for your test results.

Using predefined field names, the metadata can be added directly to your JUnit XML reports and automatically mapped to the corresponding fields in Test Management during import. Additionally, any associated icons or representations are also mapped automatically in the UI for better visualization. This documentation outlines the methods to include these fields and the field-specific behaviors.

You can add test case fields to your JUnit XML reports using the following methods:

• Property-based method
• Output tag method

Property-based method

Add fields as <property> tags within each <testcase>. This method is compatible with frameworks that support property tags. Each <property> tag includes:

• name: A predefined field name (e.g., Priority, Owner, Configurations, etc).
• value: The data as a sting for the corresponding field.

Output tag method

If your framework does not support <property> tags, metadata can be embedded in <system-out> tags using a structured format. This approach uses <![CDATA[]]> sections to include field data in the format [[PROPERTY|FieldName=FieldValue]].

Example code:

Field mapping in Test Management

The following table summarizes the predefined field names and their corresponding Test Management fields:

Field-specific requirements

Refer to the following information to understand and comply with these field-specific requirements to achieve the best results during report uploads.

Configurations

The Configurations field allows you to specify the environment in which a test case was executed, such as browser and operating system combinations. Correct handling of this field ensures accurate test reporting and analysis.

Custom fields

Custom fields allow you to include additional, user-defined metadata with your test cases, providing flexibility and context. Proper handling of custom fields ensures that all relevant information is accurately captured and mapped in Test Management.

Ensure the following for effective custom fields mapping:

• Field type validation: When using custom fields, the data you provide must match the field’s expected data type in Test Management. If there is a mismatch, the data will not be mapped, and the field will remain empty.
• Project association: Ensure that each custom field is added to Test Management and linked to the relevant project before importing test cases.

Example:

In this example:

• Data types: Ensure that the values match the data types of the custom fields:
  • Custom String Field: Should be a string (e.g., “Sample Text”).
  • Custom Integer Field: Should be an integer (e.g., 10).
  • Custom Boolean Field: Should be a boolean (e.g., true or false).
• Project Linking: Confirm that Custom String Field, Custom Integer Field, and Custom Boolean Field are defined in Test Management and linked to your project.

CI/CD References

1. Create and import test reports using Jenkins.
2. Create and import test reports using Azure DevOps.