Optimizely Achieves Testing Maturity by Moving to the Cloud

Optimizely’s on-premise grid had several limitations, rendering it ineffective.

In the beginning, the team mainly ran headless browser tests and a limited number of browser-based tests. But when they began writing end-to-end tests for real browsers, they quickly discovered how complex and unreliable these test suites were compared to unit tests for headless browsers. Real end-to-end tests were more prone to failure and non-deterministic results. Creating this end-to-end test suite, therefore, required all available engineering bandwidth for several months.

But engineering bandwidth was also required to keep the on-premise grid up and running. Moreover, the on-premise grid was inefficient in highly parallelized test environments, forcing engineers to run tests sequentially. These bottlenecks, coupled with the added overhead of building and maintaining the on-premise grid, frustrated the engineers.

In fact, frustration levels were so high that an in-house metric was invented to monitor Developer Pain Index (DPI). “DPI is a mathematical function of the time it takes to run the entire test suite, times the average success ratio. The goal is to see DPI shrinking over time,” says Brian.

The switch to BrowserStack freed up the engineering team, enabling them to accomplish their goals. “A dedicated service provider [like BrowserStack] can maintain the infrastructure better than us. Instead, we could focus on building and running tests to get better coverage,” says Brian.

And sure enough, they crafted a robust end-to-end test suite and built a highly reliable framework titled Behavely on top of a Gherkin-based system called Behave. They now have multiple CI environments running tests, and a CI/CD pipeline with multiple environments. They leveraged parallelization and other BrowserStack features to deploy bug-free releases more frequently, improving the overall QA process.

BrowserStack features—such as video and console logs—simplified the bug-detection process, boosting the overall code quality. “With BrowserStack, our engineers could perform forensic analysis or diagnosis on their tests in case something went wrong. They were empowered to deploy hotfixes to particular releases before they reached our customers. BrowserStack has enabled our engineers to perform comprehensive final reviews to confirm the quality of our code,” Brian explains.

“The move to BrowserStack has been a force multiplier across all teams. It is a core and critical part of our build and release infrastructure, and every engineer ultimately depends on BrowserStack in one way or the other,” explains Brian.

BrowserStack’s elastic cloud of browsers allowed Optimizely to test on multiple browsers in parallel, speeding up testing without any maintenance overheads or resource constraints on their CI infrastructure. As a result, the team now builds and tests much quicker.

Today, 15,000+ automated and parallelized test cases are run on Optimizely’s CI infrastructure [many of these on BrowserStack] every 45 minutes and released to a pre-production environment every 4 hours.

Efficiencies brought in by BrowserStack have lowered the Developer Pain Index by a significant margin, particularly where many browser tests would be untenable to run on an individual machine. “Because we no longer need to maintain the infrastructure, we have successfully reduced one more piece of the internal engineering puzzle,” says Brian.

With BrowserStack, Optimizely has achieved the ideal state of software testing with a high-coverage test suite, accelerated release frequency, and happier engineers.

Watch the webinar to hear the story directly from Brian Lucas, Senior Staff Software Engineer at Optimizely.