Sainsbury’s leverages test automation to double release frequency

For Sainsbury’s Groceries Online, a flawless front-end is non-negotiable. Any unnoticed UI bugs can potentially disrupt the buyer journey, negatively impacting user experience, revenue, and brand reputation in the long run. Accordingly, the team tests across a range of browsers and devices popular amongst their customers to ensure a consistent, flawless front-end.

They started off by testing manually on BrowserStack, which was sufficient for a once-a-month release cycle. However, it was difficult to scale manual testing to the newly prescribed release goals while ensuring coverage across multiple browser-device-OS combinations. It was also extremely time-consuming.

Manual regression testing for every release – every two weeks – takes almost 5 days, even when split among different testers. “It took up too much time. We couldn’t concentrate on other sprint work. It just wasn’t feasible to spend 10 days on regression a month to be able to release twice,” says Saradha Balaji, Lead Test Automation Engineer at Sainsbury’s. Besides speed, the team had other challenges too.

“We cover end-to-end testing as part of regression, so it can frustrate anyone testing the same scenario across 10-20 browsers. There is a high chance of mistakes and oversight. Testers might simply skip tests. These issues don’t happen in automation,” says Saradha. Accordingly, they started automating their tests on Cypress. But without parallelization capabilities, there wasn’t a significant saving in testing time.

Moreover, testing was assigned to QA teams on a rotational basis. If a resource who knows about regression is on leave, they had to find a new resource and spend additional time on knowledge transfer. Some scenarios required time-consuming data set up too. Soon, resource dependency and knowledge transfer also became blockers, negating the benefits of automation and slowing down releases.

Having used BrowserStack before for manual testing, the Groceries Online team also gravitated towards BrowserStack for test automation.

During the transition into the new release cadence, the team decided to go with Cypress for their automation. “With BrowserStack’s Cypress integration, we could directly start testing with our existing Cypress test suite by simply installing the CLI and configuring test run settings. We ran the suite as part of the daily deployment build and did targeted regressions during releases. Besides, we also leveraged parallel testing on BrowserStack to run tests concurrently across a range of browsers and devices,” mentions Srividya Jayaram, Test Engineering Manager at Sainsbury’s.

The team continued using BrowserStack Live to manually test one-off live issues specific to particular older devices or browser versions. When customers would report such issues, the team would simply spin up the required real device on BrowserStack, recreate the issue, and start debugging.

“The other added advantage of BrowserStack is access to Internet Explorer browsers, which we can’t do on a Mac machine. Some of our applications still need Internet Explorer support, so at least via BrowserStack, we can test on Internet Explorer,” Srividya adds.

“BrowserStack is really helpful for us to do end-to-end and regression testing around different browsers like Windows, Chrome, etc.,” says Saradha. “Before BrowserStack, the team spent 30 to 45 minutes running the current script manually on a single browser. It now takes just 10 minutes on BrowserStack.” This way, they save around 30 minutes per browser. They are also seeing the benefits of parallelization on BrowserStack. “Previously, we spent around 5-10 days testing a back-end release. Today, [with BrowserStack] we spend only a day. Even if there are issues, testing doesn’t go beyond one day,” Saradha says.

Test automation has also reduced testers’ stress and frustration. They no longer have to manually run the same regression scenarios over and over on different browsers and devices. By eliminating human error, test automation has also improved quality – which was previously at risk due to oversight. Freed up from repetitive, tedious manual testing, the QA team can now focus on getting other sprint work done to close off tickets. They now spend more time creating extra scenarios and edge cases given the regression pack doesn’t cover very detailed scenarios.

“We run a common pack before releases, which can be done by any team. This way, we don’t have to depend on anybody. It used to be a real headache before, so BrowserStack is a benefit for the entire team, not just the tester,” adds Saradha. The time spent on knowledge transfer and setting up capabilities earlier, too, is saved as testing Cypress scripts on BrowserStack doesn’t require any additional setup.

Overall, the Groceries Online team set out to speed up release cycles by going from manual testing to test automation using Cypress. BrowserStack helped the team transition to their new release cadence with added benefits on several fronts – coverage, quality, productivity, operational efficiency, and more!