Dollars lost, slow adoption, high uninstall rate, and a growing barrage of bad reviews on the app store—these are some effects of a buggy app on your bottom-line.
But these are only indicators. They won’t help you zero in on the specific causes of your app’s dismal ratings.
The reviewers will tell you that you probably ate too many lead chips as a kid—not what made them delete your app. Your analytics dashboard will tell you that the uninstall rate is going through the roof—not how to bring it back under control.
To find the bugs (before your users do), you need testers involved in every stage of an app’s development lifecycle. From design to deployment, QAs can find errors before they become costlier to fix.
In this post, we’re taking a look at all the things that QAs test (and how) during the course of a mobile app’s development lifecycle.
Prelude to Mobile App Testing: The Spec
It starts with a pen, a paper, and a question: “Is this idea feasible, or is it just another fever-dream?”
The idea becomes more realistic as you consider (and reconsider) the user problem you’re trying to solve, the market opportunity, the competition, and more. The result, if you’re lucky, is a spec—a blueprint that defines what the app/feature will do, how it will do it, which devices it will be built for, and so on.
Don’t wait till the code’s written to share the spec with the QAs. Share it with them now so they can
- understand the testability of the mobile app you’re trying to build,
- come up with better test scenarios for common and edge cases,
- set up a matrix of the right mobile devices to test on.
You’ll need them to be prepared for the stages to come.
Design and UX (Prototype)
UX and UI designers create ‘stories’ to explain the workflows in the app—and the visuals for the interface—respectively. They create wireframes based off of platform-specific usability and design guidelines. All this is to make sure the app looks and feels ‘at home’ on different mobile platforms and browsers.
QAs can help find the ‘missing steps’ in user-stories; bugs that’d turn up later as pain-points in User Acceptance Testing (UAT) stage. This helps you course-correct before development begins in earnest. Here are some common test cases:
- Are all of those app permissions necessary? Asking for app permissions that aren’t clearly necessary for your app to function will raise ‘privacy’ red-flags. Get a developer to chip in and reduce the number of app permissions you need for core functionality to work.
- Will users be able to complete simple tasks with minimal effort? If the user can’t swipe to see the next image in a gallery/slideshow, what’s the point?
- Do you need to interrupt the users when they’re trying to complete a task? If you’re showing a 30-second unskippable video ad after every move in a game of online Scrabble, just…don’t. Recall the basics of UX. Don’t make your users feel stuck. Pick up a trick or two from YouTube and show an ad that users can skip after 5 seconds.
- Will the app handle system-interrupts correctly? You’ll curse your favorite Android game (and everyone involved in making it) if it pushed an important call from “The Boss” to the background. Make sure your app knows how to handle interrupts and alerts like calls (missed and blocked), low battery, texts, and more.
- Are there too many push notifications? “Allowing an app to send you push notifications is like allowing a store clerk to grab you by the ear and drag you into their store. You’re letting someone insert a commercial into your life anytime they want.” —David Pierce, Wired.
Push notifications are good for engagement, but they can also annoy users into opting out. And once the users opt-out, your app careens downhill and joins other annoying apps in the ‘Pit of Discarded Apps’. - Can you deliver a uniform visual experience across devices/platforms? Apart from platform-specific interface guidelines, you’ll need to account for hardware differences (resolution, graphics card, color space, etc.). affect what your app looks like to end-users on real mobile devices.
By the end of it, you’ll have a prototype ready to be engineered into a working interface. The real test begins when development kicks off in sprints.
Development (Alpha)
During sprints, developers spend their time in a seemingly endless cycle of ‘code-test-fix-repeat‘. To ensure that everything they’ve made actually works, they run unit tests (typically on simulators and emulators).
Once they commit, QAs test the builds.
In this stage, QAs could be receiving smaller builds as often as twice a week (give or take a few, depending on your team and project). Note that both developers and QAs are often testing the same things (does the code work?)—just in different ways (does it work outside dev's machine?).
Here’s what QAs will be testing for:
Touch input
How does the UI respond to touch input and different gestures? Use physical mobile devices and test tappable elements, touch gestures (pinch, multi-touch, etc.), ease of one-handed usage, etc.
Accessibility
Can users complete every workflow in the app while using accessibility tools? Use:
- Espresso AccessibilityCheck class, to test every screen for accessibility on Android.
- GTXiLib, Google-developer version for XCUITest, to test for accessibility on iOS.
Notifications
When, why, and how does your app show notifications? Test:
- that error notifications show up in-app. For instance, no network, invalid input, etc.;
- how the app sends push notifications (visual, sound, vibration), how they appear on the lock screen, etc.;
- what happens if push notifications are disabled/muted.
Error handling
Start with minimizing truncation errors. Then make sure that:
- error logs are actually helpful (aka 'verbose'),
- the app can handle lost database connectivity,
- the app responds correctly to server-side and client-side errors,
- exceptions are handled and reported.
Log files
Check to make sure that your app:
- doesn't record logs by default (provide a 'preferences' screen to enable/disable this),
- log files don't exceed a reasonable size limit.
Security
This is a highly specialized subject and needs an entire post all to itself (coming soon). Generally, the idea is to protect the user-data you collect and/or share with third-party vendors through the app. Test:
- login,
- encryption,
- certificate verification,
- cookies and terminated session data (against unauthorized access), etc.
SDKs
SDKs inflate your app's size. Test to make sure they do the job you're using them for (like in-app event monitoring, third-party content, etc.).
APIs
Recall the ICE OVER MAD mnemonic (by Ash Winter). Consider who the service is for, how will they use it, the components and authentication, and so on.
App permissions
What happens to app/features if some, all, or none of the permissions are granted to it?
Interactions with system/third-party apps
Test whether your app works with system and third-party apps like keyboards, messages, calendar, map, etc.
Time and date
Recall the Candy Crush extra-lives bug? Fun times. But if users find and exploit a similar bug in your app to avoid purchases/payments, it wouldn't be too great for your bottom-line. Change the device time (to future and past) and see how it affects the logic.
Tracking
Use Charles or Fiddler to test whether your app receives the files/data it requests, without fail.
Database
Make sure database migration and changes don't affect existing data.
Consistency
Is the app in-sync with web platform? Is the app state consistent with the backend/server?
Storage
Check app behavior and try to install updates on device with full, empty, nearly full and nearly empty storage (both local and SD).
App data
Essential data should be able to store itself on the device, without hogging all the available space. Also, check whether the app data clears up on uninstall.
Battery states
Check how much battery you app drains in active and standby mode. Test how the app behaves when the battery is full, low, critically low, overheated, etc.
Standby and Multitasking
What happens to app state/user-session when your app is in the background? What if the user leaves the device untouched?
GPS and IP Geolocation
What happens to location-specific app features like language, user data permissions, blocked/available multimedia content, map interactions, etc.?
Browsers
Test your web app for compatibility on business-critical browsers. For hybrid apps, test the interactions and states of webpages that are opened in-app.
Devices and screen sizes
Test your app on iPhones and a number of Android devices (by different manufacturers) to capture UI bugs. Mix up the device sizes to make sure the frontend 'adapts-to-fit'. Here's a checklist of responsive design test cases you can run.
OS versions
Test on business-critical platforms (and platform versions) to fix inconsistencies caused by system UI, security updates, etc.
Camera, sound recorder, physical sensors, and other device capabilities
Test what happens when you deny permission for the camera, sound recorder, location information, etc. On that note, you might also want to manually test your app on physical devices in different orientations, ambient light, etc.
Performance (app and network)
Profile your app to see how much memory and processing power it uses up (when active or in standby). Test and optimize time-to-start, file requests, and workflows on 2G, 3G, 4G, Wi-Fi, etc.
Crashes
Test how the app handles crashes, memory leaks, and whether it can recover the state after abrupt shutdowns, etc.
As release day gets closer, you’ll have ‘lightly’ tested most builds. Now, you can move on to more exhaustive testing.
Stabilization
At this stage, pull requests and commits are less frequent. The possibility of tests breaking—for reasons like a randomly renamed button—is low. Now QAs can start automation testing.
QAs convert the test scenarios—most of which have already been executed in the previous stage—into ‘airtight’ test scripts which can
- provide deterministic results. That is, test executes and fails only if there is an error in app code (and not test code), and
- cover as much ground as possible—in terms of the app’s code, UI, APIs, functionalities, etc.
You have automation frameworks like Appium, Espresso, XCUITest, EarlGrey, etc. to help create and execute these tests on multiple browser/OS instances. To capture bugs that’d come up once the app is deployed in the real world, most teams test their apps on real iOS and Android devices. As builds can be tested in minutes—compared to days or weeks of manual testing—your team will be operating in a ‘Test. Fix. Repeat’ loop.
This goes on until all the bugs that could be captured with automation are fixed (or someone remembers that deadline is coming). That’s when you’re ready for exploratory testing on the beta.
Distribution (Beta testing and Release Candidate)
The beta is an ‘unfinished’ version. It means your app is functional, but whether it’s as human-usable as it can be, is to be determined.
One way to test this is with humans, as many of them as you can find. You distribute the beta to a large pool of testers. Start with your team, and after some more fixes and debugging, distribute it to participants who signed up for an early release.
This is called user acceptance or beta testing.
There are deployment tools like HockeyApp, TestFlight, etc. which help you distribute your app to your beta testers. They will manually test the app on the device/platform you equip them with. The deployment tools will let them share error logs and feedback directly with your team. This will provide the course for the next few iterations, and you repeat the process (from ideation to distribution) until you have a functional app that is very likely to delight the users.
This is what you present to stakeholders. If they approve it, you have a release candidate. Test it on real devices to do a final bug sweep.
Once you’re done, it’s time to go live with the web app. Native and hybrid apps will need to be signed and approved by the App Store/ Play Store/ marketplace of your choice. Once you get a ‘pass’, it’s time to publish the app.
If you get approved, you celebrate the win and get back to the whiteboard to improve. If you don’t, you celebrate the failure and get back to the whiteboard to improve.
Test. Fix. Repeat.
Endnote
Although the post gives this impression, a mobile app development lifecycle isn’t linear. In Agile, all of the processes discussed here take place all at once. So while the UX team is working on wireframes for a new feature, the developers are busy making a proof-of-concept. The QAs are creating new test cases or refactoring older test suites to cover new specs.
Making apps that delight users—regardless of their device, OS, or browser—takes endless abstractions, hard work, and rigorous, continuous testing. Only those committed to shipping quality code, every time, know that it’s worth it.
Recommended reading:
