Build status calculation - Unknown status in Test Observability
Understand how you can resolve Unknown status or Timeouts.
Test Observability marks tests and builds into one of the following statuses after their execution:
- Passed
- Failed
- Skipped
- Unknown
Passed, Failed, and Skipped statuses are self-explanatory. Occasionally, Test Observability marks tests and builds as Unknown, as in the following example:
Why Test Observability marks tests as Unknown
If a test is marked as Unknown, it can be due to the following reasons:
- Test Observability has not received complete data from a test due to an outdated SDK version.
- Test Observability has not recieved complete data from a test potentially due to a user-end inactivity timeout.
- Test Observability does not recognize the status that your test framework has set for the build (if it is a non standard status i.e. something other than
Pass,FailorSkipped).
If a build is marked as Unknown or has a warning tooltip, it can be due to the following reasons:
- Test Observability has not received complete data from a build or tests potentially due to an outdated SDK version.
- Test Observability has not recieved complete data from a test potentially due to a user-end inactivity timeout.
What can you do if your build was marked as unkown or had a stability issue?
-
Check if you are on the latest stable version of the BrowserStack SDK which you use to run tests with Test Observability. Upgrading to the latest stable version of the SDK for your framework minimizes issues with tests or builds being marked as
Unknown. -
If you are on the latest version of the BrowserStack SDK, then ensure you have set an appropriate Inactivity Timeout in your project’s settings. An inactivity timeout is the amount of time Test Observability continues to wait between tests. Set a large build timeout if you have large waiting periods between multiple test executions in the same build job.
-
If the issue persists despite the above actions, you can raise a support ticket.
Configuring Build Timeout
There is a possibility your build was marked as timed out because Test Observability kept waiting for a test or event from your build. You can configure an inactivity timeout to automatically terminate running builds on account of not receiving events from your test framework. Set a large build timeout if you have large waiting periods between multiple test executions in the same build job.
Updating to the latest stable SDK version
You can use the following table to update your SDK version. Upgrading to the latest stable version of the SDK for your framework minimizes issues with tests or builds being marked as Unknown.
If your framework is not listed in the table, you may assume you are on a framework which has stable versions by default. However, it is a good practice to automatically update to the latest version.
| Framework | Minimum Stable Version | How to Update SDK |
|---|---|---|
| Cypress | 1.27.0 |
Run npm update browserstack-cypress-cli in your Terminal Alternatively, set the version for browserstack-cypress-cli in your package.json as latest or ^1.27.0 and then run npm install. |
| CodeceptJS | 1.32.6 |
Run npm update browserstack-node-sdk in your Terminal Alternatively, set the version for browserstack-node-sdk in your package.json as latest or ^v1.32.6 and then run npm install. |
| CucumberJS | 1.34.8 |
Run npm update browserstack-node-sdk in your Terminal Alternatively, set the version for browserstack-node-sdk in your package.json as latest or ^1.34.8 and then run npm install. |
| Jest | 1.32.2 |
Run npm update browserstack-node-sdk in your Terminal Alternatively, set the version for browserstack-node-sdk in your package.json as latest or ^1.32.2 and then run npm install. |
| Mocha | 1.32.2 |
Run npm update browserstack-node-sdk in your Terminal Alternatively, set the version for browserstack-node-sdk in your package.json as latest or ^1.32.2 and then run npm install. |
| Nightwatch.js |
2.6.22 (for v2.x) or 3.2.0 (for v3.x) |
Run npm install @nightwatch/browserstacki in your Terminal |
| Playwright | 1.21.1 |
Run npm update browserstack-node-sdk in your Terminal Alternatively, set the version for browserstack-node-sdk in your package.json as latest or ^1.21.1 and then run npm install. |
| Pytest | 1.17.4 |
Run pip install --upgrade browserstack-sdk in your Terminal Alternatively, set the version for browserstack-sdk in your requirements.txt as >=1.17.4, and then run pip install -r requirements.txt
|
| Robot | 1.19.15 |
Run pip install --upgrade browserstack-sdk in your Terminal Alternatively, set the version for browserstack-sdk in your requirements.txt as >=1.19.15, and then run pip install -r requirements.txt
|
| TestNG or TestNG Cucumber |
1.13.10 |
Set the version of the browserstack-java-sdk in your pom.xml to LATEST or 1.13.10. |
| WebdriverIO v7 | 7.36.0 |
Run npm update @wdio/browserstack-service in your Terminal Alternatively, set the version for @wdio/browserstack-service in your package.json as latest or ^7.36.0 and then run npm install. |
| WebdriverIO v8 | 8.33.1 |
Run npm update @wdio/browserstack-service in your Terminal Alternatively, set the version for @wdio/browserstack-service in your package.json as latest or ^8.33.1 and then run npm install. |
We're sorry to hear that. Please share your feedback so we can do better
Contact our Support team for immediate help while we work on improving our docs.
We're continuously improving our docs. We'd love to know what you liked
We're sorry to hear that. Please share your feedback so we can do better
Contact our Support team for immediate help while we work on improving our docs.
We're continuously improving our docs. We'd love to know what you liked
Thank you for your valuable feedback!