How to manage Cypress Flaky Tests?

Flaky tests are tests that returns success as well as failures although there are no changes made to the code or the test. There are a lot of reasons for the test to become flaky such as environment inconsistencies, issues in the timezone, execution order etc.

You might have noticed that your build works in local, but fails when executed in CI tool. Some tests throw detached node exception, whereas some tests fail to identify element and its state or causes assertion error, because of delay in loading element. All of these could be due to a flaky design. 

In this guide, let us see how to handle Flaky tests in Cypress.

Common Mistakes That Causes Flaky Tests in Cypress

Identifying flaky tests in Cypress can be tricky. Because of the nature of flaky tests, it becomes difficult to discover the cause or reproduce the error. This becomes frustrating. That said, there are a few common mistakes that one makes which might result in the test becoming flaky.

1. Hard coded wait
2. Poorly designed selectors
3. Less understanding in retry-ability
4. Not using network wait
5. Less understanding in how the state of an element changes in application

These are the quite commonly made mistakes as seen from user complaints across platforms. 

That said, the Test Retries helps mark and track flaky Cypress tests.

Understanding retry-ability in Cypress

Cypress is smart. If you are querying for an element, it ensures the element exists in DOM before proceeding to the next chained command.

Retry Command

Let’s take an example

cy.get(“.shelf-item__buy-btn”).invoke(“text”).should(“eq”, “someText”)

Let us use the get() command of Cypress for querying the element. The get() command ensures that the element being queried exists in DOM before proceeding to invoke() command. This ensures that you don’t encounter the missing element exception when there is delay in elements loading in the application.

Do refer to Cypress’ API documentation to understand how you can use cy.get() command based on your test design.

Commands Chained with Assertion Retry

cy.get(“.shelf-item__buy-btn”).invoke(“text”).should(“eq”, “someText”)

The above example of Retry Command uses get() query command for the element and should() assertion. Cypress will first wait until the element exists in DOM which is good, but since we are checking for all the elements with the selector class matching, which is 25, there could be a delay in application rendering these elements? 

Cypress understands this scenario and retry the first immediate chained command with the assertion command. In our case it is get, cypress will retry the get() command until its length becomes 25 up-to defined timeout.

Understanding Interception To Make Tests Stable

As Cypress runs within the browser, it has total control over the network calls and can listen, modify, and wait for those calls.

To understand how you can leverage this to write our tests, let’s use the BrowserStack’s demo application

To see the network activity, Open Dev tools (Click on F12 key) > Network. You can see that BrowserStack’s demo website uses an API call to retrieve the product details.

In order to make our tests effective/stable and reliable instead of just waiting for the element state, you can wait for this API call to be successful.

it(“Intercept test”, () => {

cy.intercept(“api/products”).as(“products”);

cy.visit(“https://www.bstackdemo.com/”);

cy.wait(“@products”, {timeout: 4000} );

});

The intercept function has to be added before the line of code which triggers the api call. In our case, we need to add the API call for the product before visiting the BrowsertStack’s demo application and then wait for that call to complete successfully.

Defining Timeouts

Cypress allows users to control the timeout global/Command level which ensures the various actions complete within a given time and anything exceeding throw timeout error.

Below are the global timeouts that you can set in Cypress config file or pass it via command line

• defaultCommandTimeout

• execTimeout

• taskTimeout

• pageLoadTimeout

• requestTimeout

• responseTimeout

Detailed usage can be found here.

Summary

This article explains what causes a test design to result in flaky nature, how we can overcome the test failures due to flaky design. We have also understood the retry-ability concept of Cypress.

We have also got insight on the interception feature of Cypress and how we can use it effectively.

Whenever running Cypress tests, it is recommended to test on real devices so that the real user conditions can be taken into account for better accuracy of test results. Tools like BrowserStack Automate help QAs by providing access to 3000+ device browser combinations for a comprehensive automated testing.

Run Cypress Tests on Real Devices