Fortune 50 Business Software Case Study

Company overview

The customer is a Fortune 50 technology company that offers a wide range of business tools from productivity software to cloud infrastructure.

Their development teams operate in weekly sprints with biweekly or weekly releases, refining tools that power hundreds of thousands of businesses and are used in over 170 countries by more than 1 billion people. On its own, their cloud storage solution serves over 500 million end users

Testlio works with the team behind their cloud storage solution. They come to Testlio for enterprise-scale regression coverage that fits into their rapid-fire development sprints.

The Challenge

Given the massive product scope, diversity of end-user devices, and weekly development cadence, the client struggled to manage and execute the necessary regression testing for each release.

Potential delays due to enterprise-level product scope

Regression testing requires QA teams to comb through the existing product with each new change. The work is time-consuming – the product’s scope requires over 50 hours of test coverage each run to review in full. And there are tight windows of engineering downtime to conduct testing and deliver results without delaying development. If the speed and timing of testing don’t meet standards, sprints are delayed and critical features and fixes are not released on time.

Hundreds of end-user device/OS combinations

Similarly, the complex end-user pool of devices presents a challenge. The app runs on hundreds of different device/OS combinations. Failing to test any of these combinations runs the risk of negatively impacting the user experience in a competitive space, resulting in higher levels of customer churn.

Coordinating massive testing efforts

With thousands of testing hours across hundreds of devices, test management presents a tough logistical problem. All test cases for key devices must be conducted, triaged, and reported to the corresponding teams in alignment with the release schedule. Without strategic test management, testers may miss an area or overlap in their work, potentially causing delays or user experience flaws.

The Solution

For years, the client has partnered with Testlio to solve their regression testing challenges. Testlio provides a network of expert testers that bring devices matching the client’s end-users and deliver results quickly and seasoned engagement managers to take the logistical burden off the client’s shoulders.

Hundreds of expert testers working overnight and on weekends

The expansive product scope requires a large, on-demand, global testing team to prevent delays. Testlio leverages 162 dedicated testers in 41 time zone optimized locations to provide broad coverage overnight and on weekends. A standard run covers 60 testing hours with 30 to 35 testers per run.

Each week, the client’s engineering teams develop and commit code Monday through Friday. Then, over the weekend, Testlio’s large team of testers runs tens of hours of regression testing and delivers results on Monday morning. If testing results don’t surface any major issues on Monday, the client team can release the build. If there are issues, they can jump into another sprint to fix them.

The size and global distribution of the testing team enables comprehensive coverage in tight windows, with testing results within one to four hours and during development downtime.

Testers bring real devices corresponding to end-user devices

The client needs coverage of every OS and device used by their customers. Stocking and maintaining a device pool that covers the hundreds of device/OS combinations used by their customers could cost hundreds of thousands of dollars annually. Testlio’s networked testers bring and test on 377 real device/OS combinations. Testlio breaks their team of 162 testers into four separate but aligned units that represent device categories like Android, iOS, and web to provide efficient and comprehensive device coverage. This structure allows the client to tune the user experience and optimize performance for every customer device.

Customized fully-managed services offering

Breaking testing teams into device categories is just one of many strategies implemented as part of Testlio’s fully-managed testing services.

Each team includes a community test lead, a testing manager, and an engagement manager. The community test leads are experienced testers that provide on-the-ground QA management. The testing managers are responsible for ensuring that testing areas are covered, and that results are documented. The engagement managers oversee the overall testing strategy and output, working closely with the client to ensure that testing meets and exceeds expectations. All parties coordinate efforts on Testlio’s platform, providing complete visibility, transparency, and precise documentation for testing.

This customized, multi-faceted approach to test management allows the client team to focus on product management and development instead of time-consuming regression testing strategy and execution.

Outcomes

The client can confidently release on time knowing their regression testing is covered by Testlio across all end-user devices, supporting consistent deliveries of a world-class user experience. The end result is high app store ratings and a steady increase in customer adoption.

• On-time releases via weekend and overnight testing with results within one to four hours
• Comprehensive coverage of 377 device and OS combinations
• Upward trending iOS app store rating of 4.7 stars