At Carousell, we have six test engineers across two development centers in different locations in Asia (and we are hiring in a third!). Our experience in remote collaboration has prepared us well for the new reality of work in the COVID-19 pandemic. In this article, we will share some of the practices and tools that helped us test successfully in a remote working environment.
A day in the life of a software tester
First, let us look at some of the tasks typically performed by a tester. The sequence and priority of tasks will vary across organizations based on project requirements, team size and setup, and development culture and practices. But broadly speaking, these tasks fall into two categories:
Category 1: Tasks that can be performed independently:
- exploratory testing,
- designing and improving test cases,
- automating test cases, and
- analyzing test automation runs.
Category 2: Tasks that benefit from face-to-face interaction:
- co-working with support agents to understand customer problems,
- co-working with product managers to understand features from a user's perspective,
- co-working with developers on pair-programming, code reviews, supporting test development, etc.
The task of triaging and reproducing bugs occasionally benefits from collaboration, and so it likely falls somewhere in the middle.
Tackling testing from home
For globally distributed teams like Carousell that have adapted to remote collaboration, the challenge of category 2 tasks have always been there. Effective communication—and the right tooling to facilitate it—have helped solve these challenges. I’ll get into what’s worked for us later in the post.
But for category 1 tasks–the ones that test engineers can perform independently–the problem posed by COVID-19 restrictions is new and unique. The reason is completing these tasks require test engineers to access test infrastructure like test devices and build infrastructure. With many countries under lockdown and mandatory WFH arrangements, having these resources continuously available outside the office becomes a key factor for success.
I’m going to unpack the challenges we faced while trying to accomplish the independent testing tasks, and how we solved them while working from home.
Running mobile test automation successfully on the cloud
A critical factor in running test automation is having a reliable and well-maintained testing infrastructure. However, this is more complicated for mobile testing than for backend or web testing, as it involves physical devices (or at least simulators/emulators), and preferably, lots of them.
Maintaining such a setup on-premise can be tedious during the best of times, but without physical access to the office or test labs, it can quickly become a nightmare. The good news is that automating mobile phones or tablets can be moved to the cloud.
At Carousell, we started with a small device farm that came out of a small hackathon project, containing two Android and two iOS phones. We then added an iOS simulator farm (~30 devices) hosted on six old Macbook Pros. To save hardware resources, we had some redundancy built into the setup.
Both worked reasonably well and reduced our manual testing efforts significantly. That said, they did require occasional manual intervention. This was particularly painful before we set up VPN access, but even with the option to login remotely, not all problems could be solved without physically managing the devices. In one instance, the Macbook serving as the hub to distribute the tests to the other nodes failed on a Sunday to due VPN and Wifi issues which could not be fixed remotely. To resolve this failure, the test engineer living closest to the office had to pay a short visit to our device farm.
We continued to improve the robustness and reliability of our device farm, but ultimately decided to move the entire setup to the cloud to ensure everything can be maintained remotely. Now, for iOS, we use physical devices hosted on BrowserStack’s cloud infrastructure. For Android, we utilize a set of ~70 Android emulators running on an AWS bare metal instance (using different emulator sizes), mixing them with physical Android devices on BrowserStack based on our Google Analytics metrics which tell us which devices and operating systems our users are on.
The beauty of this approach is that everything is running on the cloud. Now, even the worst-case failures can be resolved from the comfort of our homes. Scaling our testing setup and adding new device models is also far easier this way.
Reproducing bugs on different devices
The most challenging bug for a tester is the one that cannot be easily reproduced. When this happens, you try to recreate the error on the closest approximation of the user's environment. Critical here is the device model and OS version. Android, in particular, has a sizable array of OEMs, each with different OS versions and their own unique ‘flavors’ (think OneUI or MiUI, that are basically custom skins built on top of stock Android).
For this reason, some UI bugs will only occur on a small subset of Android devices or, in the worst case, only manifest themselves on a single model/OS combination.
Our team used to keep an extensive collection of test devices at the office. While helpful, there were some challenges with that. First, our development teams are spread across three different centers across Asia (with more teams in other locations), but the majority of test devices were located in our Singapore office. More often than not, the tedious task of reproducing the irreproducible-bugs invariably fell to our test engineers in Singapore which left them with less time to automate new test scenarios or drive other test automation-related jobs.
A second challenge is the cost factor. Adding the newest flagship devices to our testing pool would quickly turn into a costly endeavor with questionable ROI.
To recreate and manually retest these irreproducible bugs, we needed access to a large pool of testing devices, on-demand. Using the devices available on BrowserStack App Live, we can reproduce a majority of user-reported issues on real mobile devices, much faster than before. Based on that success, we decided to open up access to BrowserStack devices to our development teams.
Exploratory testing on the cloud
On top of using App Live for reproducing known bugs, our teams also use it extensively for team exploratory testing sessions–what we like to call “buddy testing” and “team test parties”.
“Buddy testing” includes two engineers, preferably from different platforms (for example, one Android and one iOS engineer) who walk through a feature on an internal build to verify the changes one (or both) of them have just implemented. We found this to be particularly powerful when a change is developed for multiple platforms in the same sprint iteration. Challenges one developer faced or bugs she found during her own development are still fresh in her mind and form a strong test case to run against the implementation her colleague wrote for the other platform. On top of that, it also helps to keep the user experience consistent across all platforms.
“Team test parties” take this idea to the next level. They usually include the entire sprint team and sometimes come in the form of casual events with snacks and drinks. The participants use a mix of physical test devices and App Live devices to perform testing tours. Being able to share the screen of the test devices to the entire team and record video protocols of the tests makes App Live particularly effective.
Successful collaboration in times of COVID-19
Tools that help teams to collaborate have been around for a long time. Many of them have been designed with remote work and distributed teams in mind. In this arena, the tech industry has been prepared well for the necessities of pandemic-induced work-from-home arrangements. Video-conferencing (Zoom, Meet, Teams, Webex), collaborative document creation (Google Drive, Office 365, Confluence), planning tools (JIRA, Asana, Monday.com), you name it. For every challenge, there are plenty of solutions.
But while tooling is important, more critical is teams' ability to use them effectively to solve their specific problems. That said, it is less critical how powerful the tooling is but rather how well teams can use it to solve their specific problems effectively.
We found these two simple guidelines to really keep things moving along smoothly.
- Over-communicate and seek clarification: Information gets lost or distorted when you’re limited to phone- or video-calls for all your work communication. It’s even worse in the face of language barriers. Make a habit of seeking clarity by asking questions and repeating information even if it seems trivial. The extra time it’d take is a good investment considering the alternative: realizing downstream that half the team had ridden full-steam in the wrong direction.
- Prioritize written communication and documentation: We put more emphasis on the level of detail in all artifacts relevant to the testing: design and technical specifications, user manuals, release notes, development and release timelines, test strategy documents, test cases, etc. We make sure that all stakeholders have access to and use these artifacts. If in doubt, a written record can be a good common ground to fall back to.
For example, we would sometimes ask someone not familiar with the specific domain we are testing (not necessarily a test engineer) to review a new BDD test scenario. If they are not able to run the test, the specification is not clear enough. (There will be domain-specific terms used in the step definitions, which we explain as comments in the same feature file). Getting to that level of clarity, helps us improve the overall quality of our test suite.
Summary
We have discussed some old and a few new problems that can present themselves to testing teams when working in a remote setup or from home. We used our experience at Carousell and discussed some strategies that helped us successfully tackle these difficulties and continue to generate the highest possible impact on our mobile testing.
For our team, it turns out that the WFH-restrictions presented us with an opportunity to improve our setup to become more reliable and robust in the long-term.
