Automated tests
Pre-launch reports are automated reports that measure the quality of an app. To generate
the reports, the app goes through a wide range of tests.
The tests depict countless scenarios and interactions that are possible with an Android
device. The different Android versions are used on a large selection of simulated and
physical devices from various manufacturers.
Within just a few minutes, the tests run through the developer's entire app.
The pre-launch reports are part of the publishing process that apps go through when
uploaded to one of the various tracks ("tracks") in the Play Store.
Tracks overview
The different tracks, differ by accessibility for the test groups. Google offers four tracks here.
Link Distribution & Play Store
Company employees + selected users
Company employees + selected users
Link Distribution & Play Store
Test categories
The reports generated after the tests enlighten the developer exactly about the detected problems in the following categories:
- Stability
- Compatibility
- Performance
- Accessibility
- Security
The stability test records and captures crashes of the app. The tests are specifically
designed to find critical points where the app crashes, severely limiting the user
experience. They are performed on many different devices in order to find possible
device-dependent errors. The developer receives an overview of each test device, the
corresponding Android version, the number of crashes, a stack trace and the log messages
for the use of the app. The error descriptions are also accompanied by screenshots and
videos to make troubleshooting as efficient as possible.
The compatibility tests check all APIs that are not included in the public Android SDK.
It is determined to what extent they are compatible with the different devices and
systems. The compatibility problems are divided into three priority levels:
"Restricted", "supported but with restriction imminent" and "Unsupported, restriction
not imminent". For each incompatible API, the pre-launch report provides an explanation
of the possible incompatibility.
The tests from the performance category measure the required resources when running the
app. The test report includes the following metrics:
- Frames per second („Average frames per second“)
- CPU utilization („Average CPU“)
- Network: bytes / second sent („Average network sent“)
- Network: bytes / second received („Average network received“)
- Memory („Average memory“)
Figure 1: Test report
The metrics visualize important details about the app's performance for the developer.
The data can be retrieved as an average value for all devices or monitored on a
device-specific basis with a screen recording of usage.
The accessibility tests analyze the usability of the application. The size of the
controls and their visibility are evaluated and the usability for different user groups
is considered (e.g. for writing directions from right to left). A significant advantage
of accessibility testing is the use of multiple languages. This can reveal potential UI
problems that may arise, for example, due to longer texts in other languages.
Security tests examine the app's code for vulnerabilities that could be exploited by
attackers. The security holes found should be fixed before the app is released.
Test results
The test results of all test categories are divided into three different levels:
The most problematic results are classified as "Errors". Errors prevent the use of
the app or severely limit the user experience. The problems found should be fixed before
the app is released.
In the "Warnings" level, the medium quality problems are classified. Warnings
include problems that could make using the app difficult or prevent it in the future.
"Minor problems" cover the lowest level of categorization of the problems found.
These are mainly problems with the design of the user interface and minor problems with
the implementation (mostly just style problems). Often missing or invisible
descriptions, badly readable color contrasts and too small areas for touch elements are
criticized here.
Figure 2: Test result
Restrictions
If you want to use pre-launch reports for your project, you need to consider the following points:
- All test devices used to generate the pre-launch reports are located in the United
States. Therefore, the app may not be subject to local restrictions.
- The test devices exclusively include smartphones and tablets. Wearables, TVs and other devices are excluded from the tests.
- Apps that use a subscription model can only be tested to a limited extent, since in-app purchases are not supported by the
test scenarios.
- In addition, only native UIs can be tested. Apps that were created with OpenGL and
Unity3D can therefore not be tested. All tests are executed in portrait orientation.
