Holiday shopping this year is projected to cross a trillion dollars. US retail eCommerce spending will rise 13.2% to $135.35 billion. There’s a lot riding on seamless, cohesive eCommerce experiences this season—considering what happened during last year’s Black Friday frenzy.
Walmart, Lowe’s, Lululemon, Ulta, and J.Crew—the biggest e-tailers on the planet—went down due to high demand. The 150-minute outage cost Walmart alone about $9 million in lost sales and affected more than 6 million potential shoppers. Lowe’s’ irritated customers took to social media to complain.
To avoid all this, e-tailers double-down on the infrastructure that keeps their portals up and running. That prepares your website for exceptional volumes of traffic. But is it enough to deliver great experiences to online customers across 4 continents?
The impact of front-end bugs on online quality of experience
If your site experiences downtime, you receive alerts and everyone rushes to fix the issue till the website’s back up. Front-end bugs aren’t as ‘visible’. They don’t make the news. But their impact on the bottom line is just as harmful.
“No one wants to purchase from a website that looks like it's held together by spit and prayers.”
Black Friday is a big deal in the US, UK, and Australia, and brands have to adapt the online experience to fit the cultural context. The idea, wholesome (and profitable) as it is, involves long hours spent designing and developing UI/UX for:
- users’ device settings (language and time),
- real-time location data (via IP Geolocation or GPS),
- network connections,
- access restrictions,
- and more.
That’s a complex collection of media assets, code, business logic, APIs, 3rd-party tools and libraries in one website or app. And it can break when accessed through combinations of:
- Devices: Responsive eCommerce themes, even premium ones, are known to have broken navigation on mobile.
- Browsers: The cart-to-checkout journey, which is an eCommerce website’s bread-and-butter, can break on a major browser like Safari.
- Screen resolutions: In a usability analysis of top 60 eCommerce websites, more than half turned out to have broken product page layouts, descriptions and images on mobile.
“The number and combinations of screen sizes, browser features, input modes, and device capabilities have grown to staggering figures. Unfortunately, the testing practices and technological underpinnings of eCommerce sites haven’t been (and still aren’t) equipped to handle the onslaught of different contexts that must be supported, making layout bugs commonplace.”
—Baymard Institute (Independent UX Research Organization)
In the weeks leading up to Black Friday and Cyber Monday, engineering teams are so focused on load and performance testing that functional, UI and compatibility testing take a back seat. By the time the updates are ready for release, there’s no room for comprehensive checks and debugging. Eventually, you end up releasing buggy sites/apps to production.
So how do you deliver great online shopping experiences—on time—without cutting back on quality assurance?
QA for eCommerce in the run up to holiday shopping season
We asked our customers within the eCommerce niche to see how they deliver quality, without compromising on time-to-release. Our focus was on ways to improve visibility into release quality early in the cycle, and practical solutions that can be implemented in the short window of time (before the holiday shopping season kicks off in earnest).
So with two weeks left to go, you should:
- Let QA focus exclusively on testing: They have new automated tests to write, old tests to update, and changes to inform throughout the faster-than-usual release cycles. Anything else (like maintaining test infrastructure) is a drain on the QA team’s time.
If you are a QA engineer reading this and don’t have an in-house grid to maintain, learn how to make the most of your cloud testing solution. Check out our 12-step guide to better, faster functional testing on cloud. - Create more ‘checkpoints’ across the release pipeline. Bugs are easier (and cheaper) to fix when caught early. You don’t have to spend hours on a brand new discounts module—only to find 2 hours before release that it’s breaking the checkout page on Chrome 76-on-Android.
To prevent these surprises, distribute the testing throughout the release cycle.
Carousell, Asia’s largest eCommerce marketplace, built test suites as checkpoints for different stages of the release cycle, with a specific target (test execution time) for each. Their pipeline involves:
Regression, which comprised all automated end-to-end tests, that runs outside work hours. Execution time is not critical here, so add your exhaustive and edge cases here.
If you’re a QA reading this, check out some quick tips on creating faster regression test scripts and suites.
Sanity, which verify the stability of each release candidate or hotfix, finish within the hour or less (depending on your pace). The faster you get visibility, the more time you’ll have to debug and verify.
Fast Feedback Tests: Carousell coined this term for a suite of basic tests that verified the core functionality of their apps. They run this to verify the stability of changes before merging them. Martin Schneider, Delivery Manager, Carousell, writes, “The goal is to receive a test result no longer than 30 minutes after an engineer pushes a pull request to the Android or iOS code base. Given that it takes ~15 minutes to build the iOS app from source (Android is faster), there’s not much time left for testing.”
Apart from Fast Feedback Tests, Carousell does all their testing on real Android and iOS devices.
3. Test against as many device/browser combinations as possible. Why detract from your potential market when you can support basic purchase workflows on most of them without too much additional effort?
You can realistically aim for 70% global coverage by testing against:
- 3 device types—mobile, desktop, and tablet
- 2 iOS versions
- 5 Android versions
- 6 browser families (and specific versions within each browser family)
- 10 screen resolutions
To find out the precise combinations of devices and OS/browser to test against this holiday season, check out the Q3 2019 statistics of fragmentation across the globe. Follow it up with an explainer on how to create your own cross-device test strategy by Hylke de Jong, automation test engineer at Netherland’s largest eCommerce portal, Wehkamp.nl.
Note: Keep in mind that front-end UI tests can be slower than headless tests. To get test results within your targeted test execution time, parallelize.
Tl;dr
In 2018, well over 50% of 1,100 US-based shoppers planned to shop online during the holidays.
In 2019, eCommerce revenue during the holiday season will surpass $12 billion and most of them shopping either on the mobile or desktop.
There’s just about enough time to test your websites and apps before the big-holiday-sale floodgates open. Remember to:
- Let your QA work on test automation, instead of grid/device-farm maintenance.
- Distribute testing throughout the release cycle to avoid surprises just before release.
- Test against device/browser/OS combinations, throughout the release cycle, to prevent unexpected UI issues on production.
Go and make those sales.
Happy holidays!
Resources:
- https://www.emarketer.com/chart/222961/which-devices-do-us-holiday-shoppers-plan-use-holiday-shopping-of-respondents-by-shopping-day-sep-2018
- https://www.businessinsider.in/retail/stores-like-walmart-lowes-and-j-crew-have-a-new-nightmare-for-black-friday-and-it-signals-a-fundamental-shift-in-how-we-shop/articleshow/66779159.cms
- https://www.softwaretestingmagazine.com/knowledge/top-5-challenges-of-automation-in-retail-testing/
- https://baymard.com/blog/ecommerce-layout-bugs
- https://baymard.com/blog/handling-technical-glitches-without-losing-sales
