For non-developers, evaluating whether a mobile application has been developed well, can be tricky. Especially tricky, if you’re a product owner, product manager or in quality assurance and you need to judge the implementation from a freelancer or an external agency.
Although all software should be build robust, mobile applications usually have to deal more with flaky resources, than desktop software. This means for example: Bad network connectivity, low memory or apps getting closed by the operating system (OS). It’s easy to screw up. Those circumstances will be not obvious to inexperienced developers. Experienced developers on the other hand will account for that and give you a robust and flexible app which operates well across various devices, android versions and user behavior.
This guide explains what you should pay attention to and how you should test an android app to get a feel for its quality.
On the matter of manual tests
Why even test manually?
Testing a whole application or a new feature manually is usually not desired. In an ideal world, all your features should be covered by automated unit and ui tests. Those should be executed at least before a new version of the app is released. Desirably, they should be executed on every push to the code-base (or the develop / master branch). This way, you can be sure, that something new didn’t break something old (regression bugs) and you can test across more devices at the same time.
But most of us had already crossed projects which did not contain any or barely any tests.
There are also cases, where automating a certain test seems to costly or to complicated. For example when your app has to interact with additional hardware via Bluetooth, NFC or WiFi.
A lack of automated tests should be already your first red flag. If your developers don’t bring up the topic of automated tests at all, you should be concerned.
1. Test all the paths
When testing a feature, don’t only focus on the happy path. Try to navigate forth- and back on each screen. Cancel some actions instead of always clicking “confirm” and “OK”. You might be surprised, where developers forgot to implement a negative-path in your user story.
If your app uses run-time permissions (e.g. to access a camera), try to decline this permissions when the app asks you for them.
2. Test on different android versions
The android OS offers different functionality and limitations on different versions. Some areas get updated and changed more often than others. From my experience, you should pay special attention to this areas:
- (background) location permissions
- storage access (reading and writing files)
- running the app in the background (changes to battery saving behavior)
- access to WiFi networks from the app
This and some other functionalities require different APIs for some android versions.
Test your app with the minimum android version which you want to support as well as with the latest available version.
Also, pay attention to the release of new android versions and check your apps compatibility during the preview and beta phases. Check out if you can enroll your device for the android beta program or test your app on an emulator with the newest version.
3. Layout variations
Android devices come in a lot of different forms and shapes. Your app should look good on most of them and be at least usable on all of them. If something just doesn’t fit on a small screen, you should be able to scroll the content!
On the other hand, even if you don’t want special adjustments for tablet support, check how the app looks on a big screen. Are the icons and images still big enough? Are buttons stretched across the whole screen and look odd?
Test the app on smartphones with big and small screens. Also, test it on tablets.
Additionally, test how the app looks like when being put into the multi-window / split screen mode.
1. Open the Recents Screen menu by pressing the button on the bottom menu bar or by swiping up from the home button (depends on the device and settings).
2. In the “Recents screen” navigate to the app that you want to put into multi-window mode.
3. Hold the app-icon until a menu opens.
4. Select “Split screen”.
4. Text size
Another concern is the representation of text. Users might choose to increase the text size on their device due to accessibility reasons. Your app must account for that and still show relevant information.
In order to test this, increase the text size in your system settings and check how your app looks. Does it hide important information? Does the layout look broken?
Setting the text size (might vary for your device):
1. Go to the system settings.
2. Select “Display”
3. Select “Font size”
4. Set font to maximum size.
Last but not least, check how the app looks like when being put into the landscape mode (rotate device by 90°) or when it’s in “Dark mode” (on Android 10 and newer).
If you decided that you don’t want to support landscape mode (and therefore lock the app to portrait), at least check, that it really can’t be rotated.
The same is true for dark mode. Even if you want your app to look the same in day mode as in dark mode, make sure, that this is actually the case.
Additionally, when testing the orientation and UI mode changes, you might find other bugs, which brings us to the next chapter.
5. Lifecycle and configuration changes
All android applications are bound to a system specific lifecycle and can be stopped and destroyed by the OS at any time. This means, that apps can be “stopped” or put into the background intentionally (pressing the home button) or unintentionally (incoming call, alarm clock).
Once an app is stopped, it can be destroyed by the OS to free up resources. When this happens, the current data can and should be saved. If a user opens the app again, the previously saved data can be used to restore the user interface to it’s previous state (e.g. fill out forms again).
This is a common point of failure for inexperienced developers.
The app is also stopped and destroyed when a configuration change happens. Although this is not as bad as being destroyed by the OS to free up resources. A configuration change can be caused by actions like:
- Orientation change (rotate the device)
- UI mode change (switching to dark mode)
- (Un-) Folding up a foldable device
- Enabling / Disabling split screen mode
Test on each screen if the app still behaves as expected…
a) After a configuration change.
b) After it was destroyed to free up resources
Especially the last part is often neglected. This might lead to bugs or crashes which only appear after the app has been put to rest and opened up much later.
In order to test this behavior, you have to force the operating system to destroy your app process once it is put into the background. Luckily, we can do this through the android developer options.
Force-closing background processes:
1. Enable developer options by following this instructions.
2. In the developer options, scroll down to “Background process limit”.
3. Click on it and select “No background processes”.
With this setting enabled, you can start to test your app:
- Navigate to the screen which you want to test
- Press the home button to stop it
- Open another app and press the home button again
- Open your app again and check if it still works as expected
Make sure to re-enable the normal background process limit after you are done with testing.
6. App updates
An app that has been freshly installed can behave differently than an app which has been updated. Between versions, the schema and meaning of saved data in the app’s
SharedPreferences, files and the database might have changed. Especially databases are prone to incompatibilities if its schema has been altered between versions and no migration has been provided.
Also be aware that with the default settings, Google will enable automatic backups for your app’s data. This might lead to problems if a user deletes your app and installs a much newer version after some time. Your app must still be able to properly handle the old, backed-up data which Google automatically downloads after the app install or at least resort to a meaningful fallback behavior.
Install a new app version over an old version if the app’s data schema might have changed. Test if it works as good as a fresh install.
Testing an app properly is tedious work. As you saw in the examples, minor problems can always occur. Automated tests help your developers to spot some bugs during development but they also have their limitations.
You should know when to draw the line and to call your app “good enough” instead of trying to eliminate 100% of the bugs.