Background

Connecting Athletes Through Technology

Strava has been uniting athletes from around the world since 2009. The company was launched by digital entrepreneurs Michael Horvath and Mark Gainey as a means of recreating — on a massive scale — the camaraderie and competition they experienced as teammates on the Harvard rowing team. Today, eight years later, Strava’s mobile app and website connects tens of millions of athletes in more than 195 countries. Through Strava, runners and cyclists can record their activities, compare performance over time, compete within their community, and share the photos, stories and highlights of their activities with friends.

“Strava” is Swedish for “strive,” precisely what the company must do every day to meet the expectations of its active and engaged users. In 2016, Strava wanted to accelerate its release cycle, moving from just a handful of releases a year to a bimonthly release. To keep pace with its accelerated cycle, Strava chose Testlio as its mobile testing partner. Since then, the two have created and maintained a first-class experience for Strava’s users. “Testlio has made our user experience a competitive advantage,” says Mark Shaw, Strava’s CTO. “They ensure our app meets customers’ expectations no matter where they’re located or what device they’re on.”

“I love Testlio’s flexibility and willingness to accommodate our needs. I can ask testers to look for crashes and undesirable behaviors that we know exist but haven’t been able to reproduce on our own.”
Benjamin Woak Mobile QA Engineer, Strava
Services and Benefits

Partners in Quality

Benjamin Wolak, mobile QA engineer at Strava, says he sleeps better at night since partnering with Testlio, knowing the company is supporting Strava’s internal QA team. “I love Testlio’s flexibility and willingness to accommodate our needs,” Wolak says. “I can ask testers to look for crashes and undesirable behaviors that we know exist but haven’t been able to reproduce on our own.”

Scaling testing in-house can be difficult and time-consuming, Wolak notes. “Testlio lets us add more testers or increase the number of testing cycles whenever we want,” he says. “They also give us access to a much larger pool of testers than we could ever employ internally. These testers approach the app with fresh eyes, noticing bugs and other issues that a fatigued tester might miss.”

According to Wolak, other key benefits of Strava’s partnership with Testlio include:

Testlio better approximates Strava’s real users:

Strava’s partnership with Testlio enables its QA team to reach beyond its own unavoidable biases. “It is easy for us — as internal testers — to apply the inside knowledge that we have to justify functionality that a user would perceive as either broken or confusing,” Wolak says. “Testlio testers have no knowledge of technical limitations or compromises that have been made during development, so they better approximate how real users expect the application to work.”

Testlio tests outside of English:

Testing in non-native languages can be extremely difficult. Strava leverages Testlio for consistent testing in non-Latin character sets. As needed, Strava also asks Testlio to place an emphasis on different character sets and keyboards, or give extra attention to an area of QA that is especially challenging to handle internally.

Testlio provides wider device and OS version coverage:

Even though Strava maintains a good device library with reasonable model and OS version coverage, in practice it can be difficult for the company to ensure that they are used consistently during internal testing. “We have best practices in place for PR reviews that involve testing on older OS versions, but we still need experienced testers using less common device models.” Wolak says. “Testlio has these testers ready to go.”

Mindmap

Support for Business Objectives

Speed to Market

Another way Testlio supports Strava’s growth is by increasing the speed to market of its products. The company ships a new release every two weeks, meaning each version spends only a short two weeks in beta (through TestFlight) before going to the App Store.

“High-quality QA is essential to keeping our releases on schedule,” Shaw says. “Testlio’s experienced testers and ability to meet tight deadlines has made them invaluable to our success.”

To maintain this rigorous pace, Strava provides Testlio with a build by Wednesday of the first week of each cycle, and Testlio reports results for Strava by the end of that week. This speedy turnaround gives Strava the entire second week of the beta period to fix and test high-priority issues. “Testlio knows how important speed is to our growth,” says Wolak, who adds he can count on Testlio for efficient and thorough testing even when Strava is running late on its end. “When I send them a release, I know it will be ready in 48 hours.”

Testlio has also supported Strava on short notice, Wolak says. Although the initial scope of Testlio’s partnership with Strava didn’t cover Strava’s Apple Watch app, the company found itself in need of some added help following a recent complete rewriting of the app. “Testlio put together a team of watch testers for that with very little notice,” Wolak says.

Results in Action

Uncovering Problems Before They Become Problems

A great QA partner helps you uncover problems you’ve missed before any harm can be done. Over the course of their partnership, Testlio has helped Strava identify and correct vital issues prior to release, protecting the ever-critical quality of the user experience.

In one instance, Strava was making permissions changes for iOS 10 support. While Strava strived to cover all of its bases, Wolak also requested that Testlio provide added support by testing anything that would touch app permissions. “Sure enough, a tester found a scenario that we missed involving the camera position,” Wolak says. “Had the issue not been found prior to release, there’s a good chance Strava would have done an emergency dot-release for it.”

In another case, Strava had developed a new @ mentions feature. The team knew that it was going to have its hands full testing the feature with non-Latin character sets and keyboards. The company engaged Testlio in advance to line up Japanese, Chinese, and Russian testers for it. “The Chinese tester found a major issue where multiple mentions could be created,” Wolak says. “This was especially important because, although it would have been a major usability issue for our Chinese users, we generally don’t get as many non-English speakers contacting our support team. It might have been some time before a user reported the issue, even though a lot of people would have been having a bad experience.”

Strava can’t afford to compromise the user experience in its fast-paced, ongoing pursuit of new ways to connect athletes around the world. Thankfully, with Testlio as a partner, it doesn’t have to.