The Weather Company, an IBM business, has around 425 million monthly users worldwide. The company provides B2B and B2C users with national and local weather forecasts, reports, and severe weather coverage on digital assets such as weather.com, Weather Underground, and related mobile apps.
Till 2016, The Weather Company’s web properties mostly had desktop users. They used a combination of emulators, physical laptops and mobile devices with specific browsers and different operating systems to test their website. However, post-2016, they observed a massive shift in user behavior – their user base started using mobile devices more than desktops and laptops. Given the fragmentation in the mobile market, testing on physical mobile devices became a cumbersome, investment-heavy task. It became increasingly difficult and expensive to maintain quality and speed.
To solve these challenges, they moved testing to BrowserStack and saw improvements on multiple fronts beyond speed and quality – coverage, productivity, cost savings, and more!
Back in 2012, The Weather Company tested mostly manually for desktop users. When they acquired Weather Underground in 2013, they were introduced to BrowserStack. The Weather Company migrated some of its testing to BrowserStack, to test on different versions of IE, Safari, Chrome, Firefox, etc., in use at the time. This approach worked well for a while, and when The Weather Company was looking for a test automation solution, BrowserStack was automatically top of mind.
At the time, they were running about 500 tests on 5,000 instances a night on virtual machines. As their user base primarily used desktops, the inability and lack of infrastructure to run automated mobile tests did not bother the team.
In 2016, however, their 75:25 desktop:mobile split flipped to approximately 75% mobile and 25% desktop. A testing solution for mobile devices suddenly became a priority. An on-premise grid of mobile devices would be expensive to set up, time-consuming to maintain, and nearly impossible to scale due to the rapid developments in the mobile market.
“Even if we were to maintain an on-premise device farm of only the top 10 user devices, we would have to spend upwards of $50,000 a year, and we would still not have any automation capabilities out of the box,” explains Todd Eaton, Head of Consumer Product DevOps & QA at The Weather Company.
They turned to BrowserStack again to see if the platform could also handle automated mobile tests cost-effectively. A few POCs later, BrowserStack became the preferred solution for testing at The Weather Company.
“We felt BrowserStack would be the perfect solution for our use case. It allowed us to mimic our users’ real-world environments on BrowserStack’s real devices. We could use our test scripts on BrowserStack to run automated tests on various iOS and Android devices, different versions of browsers, etc., to deliver the quality our users expect,” says Todd Eaton.
It took the team some time to fine-tune their BrowserStack usage and get their error rates down but support from BrowserStack made it seem effortless. “Within 6 months, we figured out how to get the optimal throughput for our overnight testing and get accurate results with low error rates.”
Additionally, the team has a great working relationship with BrowserStack. “We have shared many enhancement and feature requests and have seen those improvements come through. We were also one of the beta teams trying out visual testing when BrowserStack was evaluating Percy for acquisition. We give BrowserStack candid feedback and see the suggestions being taken seriously,” he adds.
Today, The Weather Company is the single largest BrowserStack user within IBM. They went from 50 Live licenses to 75 and from 5 App Live licenses to 30. They use all 5 BrowserStack products – Live and App Live for manual testing, Automate and App Automate for test automation, and Percy for visual testing.
“Automated tests on BrowserStack cover a lot of our workflow, but the bugs we see these days are UI based. As the website and app evolve, with more unit tests and assertion tests in place, it becomes difficult for automated tests to pick up UI issues. That is where Percy helps us a lot.”
With BrowserStack, The Weather Company can effortlessly deliver the quality users expect. “BrowserStack makes it a lot easier to test our products against all our end users’ browsers and devices, and make sure our products work perfectly. Team members can access BrowserStack from anywhere and test on any device they want. No need to search for physical devices, set them up, or tear them down after tests. It gives us a lot of flexibility,” says Todd.
The team is seeing speed improvements too. Todd explains, “Thanks to parallelization, we get through 20,000 tests in a couple of hours on BrowserStack. This helps us push critical features that people might want to use for an upcoming snowstorm or a hurricane much faster than we could if we used VMs or an in-house device lab. It saves us a lot of time, money, and effort.”
Features built into BrowserStack that can simulate different network conditions such as bandwidth, latency, geolocation, etc., help the team account for all their users. “Millions rely on The Weather Compnay; it is a crucial part of their lives. We want to give them the most accurate forecast of inclement, perhaps dangerous, weather coming their way. Our nightly tests on BrowserStack give us the confidence we need to push important releases a couple of times every day. It significantly benefits our end-users.”
Join over 6M developers & 50K teams across 135 countries.
