Under global lockdowns, as software teams across the world try to maintain productivity and ship quality releases at speed, how you test matters more than ever. So in this checklist, we are sharing thirteen optimizations to make automated functional testing on the cloud faster, less-flaky, and overall more efficient. Added along the way are resources and examples to help you better understand and implement these best practices today.
1. Target an acceptable test execution time
Target practical execution times for different automated test suites
Optimize the number of tests—and their execution time—for different checkpoints in the pipeline. For inspiration, see how Carousell, Asia’s largest online marketplace, split up their test suite to distribute testing throughout the pipeline and release daily, with confidence.
2. Test on every pull request (if not every commit)
Run functional tests at every pull request
Prevent testing bottlenecks near release days by isolating functional bugs early in the lifecycle. Aim to run functional tests after every commit. If not, run them on every pull request.
Watch: The Agile testing and release process at IBM-owned weather.com
3. Configure your CI setup for efficiency
Set up your CI based on the frequency of testing, projects tested in parallel, and the number of tests to be run
Use a master-slave configuration in CI such that the master machine can handle dispatching and monitoring jobs, and reporting the results—all without overloading or slowing down.
Watch: A step-by-step walkthrough on building an efficient, well-integrated CI pipeline in BrowserStack Summer of Learning, Episode 3.
4. Keep your tests short
Shorter tests run faster and are easier to maintain at scale
Improve your tests' execution time and flakiness by writing atomic tests. Use Selenium requests sparingly to minimize the number of network calls.
Read: How to write Selenium tests that run faster in our 3-part guide to faster regression testing.
5. Execute your tests in parallel
Run tests concurrently instead of sequentially for faster builds
Speed up build times by 10x or more: by configuring your framework to initiate multiple tests simultaneously.
Watch: David Burns, contributor to the W3C WebDriver specification explains how to correctly harness the power of parallelization in Selenium.
6. Prevent queuing with ‘blending’
Avoid hitting tier limits and improve build times
To prevent DDoS attacks and abuse, cloud testing platforms limit the maximum number of parallel tests you can run on a specific browser/ device at a time. Once you hit this limit, the rest of your tests will be queued.
Find these limitations and distribute your tests between browsers and devices to prevent queuing and speed up testing.
Learn more about desktop and device tiers on BrowserStack.
7. Use built-in debugging options
Make debugging and collaboration easier by adding context to bugs easily
Built-in debugging tools help you isolate bugs and their causes. Instead of building your own logic and saving it locally, use the debugging features and REST API offered by your cloud platform to make life easier.
Learn more about built-in debugging tools in BrowserStack Automate and App Automate.
8. Test on the right browsers
Test on browser versions that are a.) popular, and b.) problematic
Despite W3C's standards, each browser family (and versions within the family) have different ways in which they render your website to end-users. We covered these differences in a previous blog post about the reasons why you need cross-browser testing for compatibility.
To make sure your UX doesn't break from browser fragmentation, test on browser versions used by at least 80% of your visitors and your target market.
If you are using Google Analytics, go to Audience > Technology > Browser & OS and pick the browser versions that have the highest number of sessions and/or the highest bounce rate.
To see browsers that are popular in different countries, check out this resource about different stages of fragmentation across the globe from our fragmentation series.
9. Test on different screen sizes
Test on multiple screen sizes, viewports and resolutions
Different screen sizes affect how your website appears to users. Ensure that your pages are responsive and that all core and business-critical functionalities render perfectly on different screen resolutions.
Read: The ultimate responsive design checklist.
10. Test on real mobile devices
Test against physical mobile devices for conclusive results
Over 50% of web traffic is on mobile devices. To make sure that the UX doesn't break on mobile, teams rely on emulators and simulators for testing. But while they are good for early stage unit testing and prototyping, they can never provide conclusive, reliable test results across real world scenarios. Incompatibility with devices and browsers is also among the top reasons for revenue leaks in the funnel–making compatibility testing essential to your end-to-end testing strategy.
However, given the state of fragmentation in the mobile market, it'd be impossible to test against every device in the market. For a more realistic test bed and reliable results, test against the most popular devices within your addressable market. Use your own site analytics to determine popular devices among visitors. You can also use crash data as valuable test input, as WehKamp does.
Find popular devices in your target market with our test on the right mobile devices report.
11. Test in real user conditions
Recreate real-world user context—and test against them
Variables like network connectivity, GPS geolocation, pre-existing apps, device languages can introduce bugs. Configure your test bed to imitate real-world scenarios to minimize the 'edge-cases'.
Learn more about testing in real user conditions.
12. Set up alerting, monitoring and bug tracking
Set up real-time alerts to notify you when a test/build fails
Maximize testing and debugging efficiency with real-time notifications about your build status.
Check out: The Testers' Toolkit: BrowserStack integrations to keep you productive.
13. Keep your test hygiene in check
Dynamically name your builds and test runs to identify failures easily
Ensure that you dynamically name your builds and test runs using Environment Variables in your CI/CD tools. It makes isolating errors easier.
You should also mark your test status (Pass/Fail) with our REST API. While testing on BrowserStack Cloud, these practices ensure that you get accurate and useful information on the Test Insights dashboard.
When there's no way to gain access to or troubleshoot your in-house grid of devices/browsers, the cloud can save QA. It is continuously available to execute your tests. Teams that embrace cloud testing strengthen their test and release pipelines, making the lockdown-induced dent in productivity and release velocity minimal at best.
We hope this list of optimizations helps you test better and make the most of your cloud testing platform.
If you're considering the move to cloud, be sure to consider factors like latency and reliability while you compare different vendors. Our solutions experts are available to answer any questions.
Enable your team to test anytime, from anywhere with cloud testing on BrowserStack. Try it for free.
