Data Quality Testing Case Study

The Company

The customer is a Big 6 broadcast media corporation that produces and distributes sports, news, and entertainment content. They have over 20 apps where users can stream programs and receive local news updates. Their news app alone has eight million monthly recurring users.

Testlio worked with the data strategy unit responsible for making data-driven recommendations on programming and product optimization. Their data comes from calls to data hosts like Google Analytics and Adobe Analytics. Despite their broad scope, they do not have an internal QA team to check that these data calls are accurate, and their previous crowdsourced testing experience produced subpar results.

The Challenge

Consumers are emotionally connected to the media they consume, and major programming decisions like discontinuing a show can have a big impact on brand sentiment. Even changing app navigation can negatively affect user experiences. It is critical the data strategy team has accurate information, from viewership counts to in-app behavior, to drive their recommendations.

Yet app updates frequently break links to the analytics platforms that track viewership and behavior metrics. The client needed a way to verify that calls to these platforms fire and produce accurate data. They also needed to account for the challenging scale, complexity and dynamic nature of their app suite.

Massive Scale: Over 12,000 data points to verify

Data strategy units tend to be small and often lack internal QA expertise. The client team includes only a handful of data strategy managers and directors without any testing resources. At the same time, their scope of work is huge, covering more than 12,000 data points per run from over 20 apps running on multiple platforms like Roku, Apple TV, and mobile.

An issue with any one of these 12,000 data points compromises the team’s ability to make sound recommendations. For example, if Adobe Analytics isn’t registering the call that signals buffering video, they won’t know to recommend a video player update to fix the buffer. In turn, the affected streaming app will continue to lose viewers and brand loyalty until the bug is fixed.

To make accurate recommendations, the team needed a testing resource to catch issues on every call, for every app, on every platform.

Complex Testing Process: Diverse call specs and requirements for each app

Another challenge is call specs must be accurate to a granular level. The data strategy team cannot make sound recommendations if testing doesn’t surface hard-to-catch errors. For example, the call signaling unique viewership, “Video_Start,” might include a syntax error and register as “video_start.” The error causes an undercount of unique viewers. Based on this flawed information, the data team might recommend discontinuing the affected content, even though it is actually a more popular program.

Adding to the complexity of testing, each app’s product team tracks metrics differently and has different testing priorities. Their previous crowd testing vendor leveraged new and often inexperienced testers to participate in each run. Testers that are unfamiliar with an app’s calls or priorities often miss issues or report low priority bugs. This leads to inaccurate recommendations and the client spending valuable time sifting through bugs, or time and money spent training the vendor’s testers.

Given their small team and large workload, the client needed a testing partner that could handle the complexity of testing without requiring oversight.

Changing Needs: Varying focus areas from month-to-month across over 20 apps

The client’s suite of apps is constantly evolving with frequent updates, new app rollouts, and cancellations of underperforming apps. Testing requirements change each month to match new priorities.

Their previous crowd testing vendor was not flexible when requirements changed. They had a hard cap on hours and would stop testing in moments of peak demand, forcing the client to deal with missed bugs and inaccurate data, wait for testing hours to refresh, or negotiate a new contract.

The client needed a testing resource that could adjust to evolving requirements.

The Solution

Testlio provides a fully-managed analytics testing service that addresses the client’s unique challenges with a scalable network of testers, dedicated testers for run-over-run consistency, and a flexible service agreement.

Strategic vision supported by a burstable network

Without internal QA resources, the client needed a comprehensive testing strategy as well as test execution that wouldn’t tax the core team. Testlio is able to provide both the process and the people to make that happen.

Before testing begins, the client sends Testlio’s engagement managers a list of call specifications. The engagement managers translate these specs into a comprehensive plan that covers the entire scope of testing. Then, they assign roles to dedicated testers – one tester might cover video playback for live content, while another might cover short-form content – to prevent overlapping workflows. Each run includes a scalable group of testers in burstable teams of 10-50 to swarm the testing surface and deliver results, fast.

Testlio’s ability to translate specs into comprehensive test plans, coordinate a large team, and scale execution to meet demand provides fast, robust testing coverage, even during peak demand.

Dedicated testers create confidence from complexity

The client’s previous crowd testing vendor brought different testers for each run who were prone to missing issues. Testlio takes a different approach by leveraging expert, dedicated testers.

Project over project, Testlio leverages dedicated testers instead of different testers for each run. Consistent testers understand the nuances of calls, how to streamline reports, and which issues to flag for different product teams. They improve with each run and are less likely to miss issues.

The client trusts Testlio to fully manage complex testing efforts, which frees up their time to make strategic and impactful recommendations to internal stakeholders.

Flexible partnership when priorities change

The client’s varying release schedules caused problems for their previous crowd testing vendor, including a testing shut down during peak demand. Testlio’s networked testing model is structured for flexibility and supports a more responsive partnership.

Testlio provides a bucket of 900 hours per month. Within this allotment, the client can shift focus areas. For example, the client can conduct a full analytics sweep on six apps one month, then tackle eight different apps the next. When the client is on pace to exceed 900 hours of testing, Testlio provides an early warning and can apply rollover hours under a flexible partnership agreement rather than shutting down coverage.

Between these two structural advantages, the client can now count on a flexible partnership that allows them to shift testing areas, priorities, and scale testing up or down without risking data quality or delaying releases.

Outcomes

Today, Testlio discovers over 150 critical data integrity issues for the client each quarter, providing confidence that their data is accurate in order to make data-driven product and programming decisions.

• Full coverage across more than 20 apps on multiple platforms
• Strategic testing management provides time back for the internal team
• The flexibility to change coverage without creating contractual or resource bottlenecks
• $240k saved annually compared to what the client would pay for an equivalent in-house team