Quality Assurance (QA) is critical to delivering brilliant customer experiences. But, with a plethora of types of testing types and methods, testing a mobile application for release can be overwhelming. To get you through it, we’ve outlined the essential mobile application process below.
Intro to Agile Testing and Continuous Integration
According to an Atlassian survey, 80-90% of DevOps teams today use Agile methods. Agile development has had a huge impact on Quality Assurance, requiring QA teams to test faster and more often. With an average sprint lasting 10 days, this means mobile app testing and preparation occurs throughout the entire development lifecycle. QA teams usually need to write test cases, while Dev teams are developing so that testing can happen in a short window at the end of the sprint.
Another critical trend impacting application testing is Continuous Integration and Continuous Delivery (CI/CD). Compared to Agile, CI is more of an iterative approach, in which code is developed in small, continuous updates and releases. Rather than testing an entire mobile application in a sprint, QA teams can test portions of a feature, or maybe 2 out of the 5 features done in that sprint. As code is produced in smaller segments, it is tested. End-to-end testing usually occurs prior to a release at the end of the sprint.
These trends require mobile application testing to happen faster than ever before, placing a greater need for speed and efficiency on QA teams. To keep on track, it’s critical to maintain a strategic mobile application testing process.
1. Test scope & coverage
The first step is to determine what you will test, specifically the features and functionality of the app. Previously, QA teams often created a test strategy, big-picture document which outlined the goals and methods of the testing process. The Agile framework, however, advocates for “working software over comprehensive documentation.” Thus, your testing strategy may not be formally documented, but it is still critical to clarify.
The most important step of a mobile app testing strategy is defining what features and functional requirements you are testing. This will determine the types of mobile app testing required, whether functional testing, usability, compatibility, performance, security, or others.
Likely, your testing scope will include a mix of functional and usability testing. Some of the most common functional features to test are the sign-up and log-in experience; performance across different connections and carriers; and checking layout on different screen sizes. For other factors to consider, see our check-list: 9-step mobile app testing strategy checklist.
While every mobile application has some basic functional testing needs, consider consider the specifics of your app as well when defining your scope:
- Does the app interact with other apps?
- Is the app Native or Mobile-web or Hybrid?
- Is the app testing include only front-end or back-end testing as well?
- Is it compatible with multiple networks?
Identify Testing Locations
For apps with a global user base, one notable type of testing is localization testing, which tests your app in a variety of locations and languages. This goes beyond just using a basic translation tool, and ensures your app avoids common language or cultural mishaps. Typically, localization testing is done in partnership with QA company with a global network of testers to test the app in the locations of your users.
RELATED: See how Strava leverages Testlio’s network of native speakers to deliver a brilliant customer experience across the globe
Decide Device Coverage
You’ll also need to decide which, and how many, target devices to test on. Our research has shown leading mobile app companies test over 24 device-OS combinations prior to a release. Smaller QA teams may take a more incremental approach, testing an app on a handful of devices. When selecting these, again, consider the specifics of your app:
- What operating systems does your app support?
- What are the earliest versions of the relevant OS?
- What are the most popular mobile devices among your target audience?
2. Plan mobile app tests
Once you know your testing scope and coverage, the next step is to determine how to test. In other words, you’ll need to break the strategy down into a plan of action, detailing who / what will execute your tests and on what timeline. An essential component of this step is preparing test suites: writing test cases or test scripts for automated testing.
Manual v Automated Testing
Today many QA teams integrate automated and manual testing, as each has different pros and cons. Generally, automated testing is useful to verify repeated tasks and solid code (e.g. verifying the throughput of an API, or scale testing). The drawback is that writing and updating the test scripts to verify an app’s standard features can be time consuming.
Furthermore, automated scripts cannot substitute for ‘in the wild’ testing. Sure, the UI may function great in the lab, but does it work while a user is walking down a busy street holding a device in one hand? Delivering a brilliant customer experience in real-world settings is what sets 5-star apps apart.
Because of the drawbacks of automated testing, most mobile applications still rely on manual testing in some form. To combine the best of both worlds, many QA teams integrate both automated and manual testing for different stages of the software development life cycle.
Write Test Cases
Depending on your testing method, you’ll need to write manual test cases and automated test scripts. This can be done in-house or in partnership with a QA company. If writing yourself, check out this tutorial on how to write a test case. Or, if looking to accelerate your QA, check out Testlio’s ‘do it for me’ approach, which includes test case design.
Identify Testing Approach / Partner
If you have limited testing needs, you can execute test cases/scripts in-house with your own engineers, but this has some drawbacks. First, scaling is difficult. Software testing is usually needed very quickly, and for short periods. Covering these ‘bursts’ of demand is challenging.
More importantly, testing in a lab environment is very different than testing ‘in the wild’, where usability or functional defects often occur. This can lead to poor customer experiences or, in some cases, high-profile failures. (E.g. the app used for the 2020 Iowa Democratic caucuses failed very publicly due to a lack of usability testing with real-life testers.)
The limitations of in-house testing have led many mobile app companies to use a crowdsourced QA company. While crowdsourced testing has historically had some disadvantages, these can be mitigated by selecting the right QA company (notably one with highly-trained and motivated testers). With the right QA partner, crowdsourced testing can augment your current testing efforts with scalable, high-quality testing.
When selecting testers (either with a crowdsourced QA company, or on your own), it is advantageous to use testers who are similar to your users. For example, if you have a technical mobile application for a business customer, you’ll ideally want testers with domain expertise who can navigate and test it properly. Some crowdsourced QA companies can curate this specialty tester network for you.
RELATED: The QA Partner Playbook
3. Execute app testing
Now you’ve identified and created the test cases and scripts. The next step is to run these. The specifics of executing tests depend on your test scope and coverage, and release cadence, which vary widely from company to company. Most DevOps teams release daily, weekly or monthly.
To give one example of executing a test plan, let’s use an example of one of Testlio’s clients, a large TV network with a mobile application. The app is updated and tested daily, and they use a continuous integration (CI) approach. Specifically, every evening, the CI system sends the build to a device farm which runs and manages the automated test suites. When the test results are ready, the automated suite sends the results back to the client.
In parallel to the automated tests, the client also uses manual testing using Testlio’s network of testers. The CI system sends the build to the Testlio platform, then the testers start their test cycle and run it overnight. With both the automated tests running in the cloud, and Testlio managing further usability and functionality testing with human testers, the tests run overnight and provide results in the morning.
This example presents one extreme: a fast release cycle using continuous integration with automated and manual testing. Naturally, testing cycles can be longer and simpler, depending on your app testing requirements.
4. Track and manage defects
After test runs are completed, you will need to manage and correct defects. The key here is correctly prioritizing defects, so that you can address the most important things first.
Defects can be tracked in a project management software like JIRA. If outsourcing app testing, your QA partner will likely report bugs directly to the project management system as well. Common defect priorities include:
- Low: no major impact to business functionality (e.g. a misspelling in the UI)
- Medium: functionality is not working as expected, but there is a work around
- High: critical functionality is blocked, but there is a work around
- Blocker: critical business functionality is blocked and the user has no work around
The correct prioritization of defects is a key point in managing your app testing process. Many times, you will be inundated with low priority defects, such as a misspelling, making it harder to find and target crucial defects, such as the app crashing.
This is where making strategic choices about your Quality Assurance partners can have a major impact. At Testlio, testers are paid by the hour, not by the bug, which leads to higher quality issue detection.
RELATED: See how SAP Concur solved 3x more issues in ½ the time using Testlio’s experienced tester network.
Finally, review the mobile application testing process after the test run is complete. One option is to create a test summary report. Regardless of style, it’s important to document:
- Which devices and OS versions were tested
- A review of the tests performed, and major defects uncovered
- How many tests passed and failed
- If appropriate, you can also make a final recommendation on if the mobile application meets all the acceptance criteria and is ready for release to the public.
If you have gotten this far, you’ve probably gleaned that mobile application testing in an agile environment can be challenging. Succeeding requires strategizing, planning and tailoring these steps to your mobile app.