Leading Sports App Case Study

The Company

Our client is the sports network arm of a leading publicly traded media company. The network produces a leading app and website for multi-sport streaming, score updates, and news. Their employees worked tirelessly to produce content for 100+ million viewers last year. The hard work paid off as the sports network has won many major awards, including multiple Emmys. Additionally, their mobile app ranks in the top-15 on the Apple App Store and Google Play.

With a delayed product launch and a shortage of in-house testers, the sports network turned to Testlio. We delivered web and mobile app testing services to identify livestream bugs, faulty APIs, and complicated user flows. Together, Testlio and the client released a world-class mobile application and website update contributing to a 22% increase in their App Store rating.

Today, Testlio’s distributed and experienced testers flex in and out of assignments for the sports network and other divisions within the broader media organization. Testlio also helps with launches, market updates, significant events, and more.

The Challenge

Sports fans are some of the most passionate consumers in the world. However, they can become a cantankerous lot when they can’t easily watch sports, update themselves on scores, or read articles on their favorite teams.

Our client needed to address bugs in their video streaming experience and news delivery to not only prevent customer churn, but grow their audience impression count and associated advertising revenue. A significant update without these fixes was out of the question.

Their Challenge?

After some starts and restarts, the network was behind schedule; but wouldn’t release their streaming and sports news service until it offered a seamless user experience on a full range of devices. And they couldn’t preemptively test for that experience without additional resources. An added twist to the release was the onset of the COVID pandemic. The in-house testing team now required to work from home lost access to the company device lab.

While automated testing was a part of the mix, several user flows were too complicated to automate. The sports network needed a testing partner with a deep bench to augment their in-house QA team and fill the gaps missed by automation.

What was wrong with the pre-release site, application, and testing process:

• Faulty APIs that funnel the entire app experience negatively impacted ad revenue and data strategy.
• Varied elements, design, content, and interactions across platforms and multiple devices led to usability challenges for the end-user.
• Certain video stream buttons weren’t working.
• The in-house testing team had finite bandwidth and access to devices required to match the speed of the agile development process.
• The team also needed help writing automated and manual test cases and performing overnight regression, exploratory, compatibility testing and bug verification.

The Solution

Full Coverage Testing for User Experience

The client needed a consistent user experience on all platforms. To that end, Testlio conducted a wide range of UI, performance, data compatibility, content, and audio/video testing across dozens of browser, OS, and device types. For example, testers explored every available application screen to make sure that the installation worked. They also ensured that there weren’t any UI issues due to device screen size or device-specific compatibility issues. Similarly, testers interacted with every possible screen element to measure user flow impact.

Detecting Faulty APIs that Funnel the Entire Experience

Testlio also conducted a particular type of exploratory testing to crawl the entire site manually. This exercise revealed various data issues from faulty API’s and provided extra confidence in the release’s stability.

Livestream Testing

Streaming services manage vast amounts of data, and a single bug can collapse the entire stream. In the network’s case, they needed to quickly identify and fix issues to ensure a consistent streaming experience and build trust that the experience was perfect before this critical product launch. Testlio conducted livestream testing in the top designated market areas (DMAs) to assure streaming quality and geo-specific data transfer.

Networked Testing – Engagement Managers

The Testlio engagement team sets the foundation to test. With the help of our global network of on-demand testers, we then tackle broad, high-stakes, agile testing challenges. For example, the client’s engineering teams would develop rapidly during the day. With a log of written test cases to reference and as bugs emerged, Testlio testers would spin-up on-demand, testing in short bursts – 20-30 testers at a time – and across time zones – so that teams could alternate and work 24-hours a day.

The beauty of this arrangement meant that when the client’s developers, based in North America, went home, Testlio teams continued to work, test, and document the user experience. And because Testlio testers own 1000s of devices with the right mix of OS configurations, they were able to provide full coverage quickly. It’s an example of what happens when aligning agile QA with agile software development.

Networked Testing – Engineering Alignment

Testing and quality assurance initiatives must be tightly aligned with product and engineering teams. The seamless sync between Testlio’s software and the client’s existing test management tools expedited delivery of the real-time bug and user experience updates. Without changing their preferred workflows, the client’s engineering team performed quick fixes by gaining immediate visibility into the debugging efforts. For example, Testlio delivered vetted and highly filtered reports to the client overnight. This meant that, upon return to the office, engineers could begin addressing fixes immediately and not waste time questioning the quality or how to prioritize the bugs.

Outcomes

With the help of Testlio, our client gained the confidence needed to make a major feature update and release. This confidence came through regular, comprehensive test coverage. They now release to production with the knowledge that they have had experienced and consistent testers vetting their apps for user experience and compatibility issues. Replicating Testlio’s efforts in-house would require a significant financial and time investment by their development team. Besides, even with additional full-time personnel, the client may not have the same level of flexibility or benefit from the different perspectives Testlio offers. What’s more, access to 1000s of devices and OS combinations is another challenge for in-house teams, even before COVID-19.

Key outcomes:

• Platform-wide usability and flawless API functionality for a high-stakes product launch
• A fully functional streaming experience
• Accelerated release after months of delays
• 22% increase in app-store ratings from the previous version

With a highly selective testing vetting process, client NPS score of 71, and a 4.8/5 star rating on G2, Testlio provides an industry-leading approach for clients’ holistic web and mobile app testing needs.