Request info

SAP Concur Case Study

SAP Concur 6x’s find rate with Testlio

3x issues

resolved

50% decrease

in time spent resolving issues

77 issues

found last quarter

The Company

SAP Concur is an industry leader in cloud-based travel and expense-management services, offering scalable solutions designed to help employees stay focused on results—not receipts.

With an expansive portfolio of complex B2B integrations, QA is a critical part of the agile development cycle. Crowdtesting gives SAP Concur the flexibility and coverage it needs—but as it learned, not all crowds are created equal.

Testing areas

Exploratory, functional, performance

Release cadence

Daily

Industry

eCommerce

Stage

Global expansion

The Challenge

When George Andraws left Apple in 2016 to lead the QA teams at SAP Concur, he knew there would be challenges. Despite providing innovative expense-tracking solutions for today’s top firms including Uber and Airbnb, “we weren’t thinking ahead,” says George. “It was random, not well-organized and reactive … a discorded effort.”

At the time, SAP Concur was relying on pay-per-bug testing from a crowdsourced marketplace for regression testing. George says he was hopeful the model would prove motivational for the testers; however, it soon became clear that something wasn’t working.

“With our other vendor, there wasn’t a great focus on testing and quality. I tried to work with them … but we didn’t see the results we needed.”

George Andraws, Senior Manager, Quality Engineering at
SAP Concur

“They weren’t finding bugs that we were actually going to fix,” says George. “It was bug-finding, not customer-issue finding. Sure they found a lot of bugs, but a lot of them were very extreme edge-cases … there wasn’t a great focus on testing and quality.”

As the lead QA manager, George tried for months to improve the ROI, but nothing worked. “The problem,” explains George, “is that when you deal with enterprise and money and travel, things get very complicated very quickly because companies want a lot of control over the configuration. It just doesn’t work to have testers who are constantly rotating, who are more random and just follow test cases.”

“There came a point when you’d get an eye-roll every time you saw one of their bugs. It was ‘Oh no, here’s another 5 minutes of my time wasted looking into a bug we’re probably not going to fix.”

George Andraws, Senior Manager, Quality Engineering at
SAP Concur

Device/ OS combos

119

Locations

25

Test report turnaround

Next day

Integrations

Jira, Zephyr

The Testlio Difference

Concur moved on from their original vendor in 2016 and partnered with Testlio, opting for professional QA testers with dedicated, full-service teams at scale.

“Testlio’s model is different. They focus on actual testing, and finding bugs is a by-product of their great testing.”

George Andraws, Senior Manager, Quality Engineering at
SAP Concur

Since then, the results speak for themselves. Crunching the numbers reveals both total reported and resolved bugs have tripled since partnering with Testlio—and in just half the time.

“The find rate went up by 6 times. The fix rate improved. And when you look at the total number of bugs fixed.
That’s huge. That speaks volumes.”

George Andraws, Senior Manager, Quality Engineering at
SAP Concur

George credits Testlio’s dedicated-tester model, especially his point-person Rudolf whose holistic product knowledge now rivals even his own. “Rudolf has been fantastic. He knows how everything works together, so he’s able to take input from one team and see how it would impact another. He’s always thinking about quality.”

While working with pay-per-bug crowd services may offer adequate coverage, when it comes to quality and outside-the-box solutions they often fall short.

The difference was particularly apparent during one heroic effort, when a high-stakes hotfix was needed within 24 hours over a weekend. Without fail, Rudolf developed and seamlessly executed a new functional testing solution, avoiding possible service disruption and keeping customers happy.

“It really has been fantastic working with Rudolf. He is very proactive. Very thoughtful. Detail-oriented.
It’s like having someone who really cares about the quality of the product and how to test it better. That is really rare to find in a vendor.”

George Andraws, Senior Manager, Quality Engineering at
SAP Concur

QA is now an integral part of SAP Concur’s development cycle and its customer success. They are embedded in the agile and scrum teams and involved in most of the process. “We try to get involved earlier. It’s not a throw-it-over-the-wall model,” says George.

One of the biggest differences George notices with Testlio is how it helps him be more proactive, less reactive. “Testlio asks questions before we even think about them ourselves,” says George, whose teams are already one step ahead of both iOS 12 and Android P launches thanks to Rudolf.

And finally, when it comes to dynamic decision-making, it helps to have all your key data in one place. George appreciates Testlio’s comprehensive, user-focused platform and how test results integrate seamlessly with SAP Concur’s native tools of Jira + Zephyr.

“This has been an exceptional experience working with a third-party test vendor like Testlio.
There’s a personal touch, and you see that not only with Rudolf, but also with the rest of the Testlio family.”

George Andraws, Senior Manager, Quality Engineering at
SAP Concur

Case Studies

…

Royal Society of Chemistry Publishing Case Study

Not-for-profit, academic publisher runs accessibility audits and solves 17 critical WCAG compliance issues.

…

Making Space for Testing at Ricoh Spaces

Rapidly scaling ‘startup-within-an-enterprise’ learns to sprint with Testlio. Learn how they did it in this case study.

Testlio client LGND.art…

LGND Case Study

LGND.art cut testing time from 20 hours to 15 minutes and built a true CI/CD pipeline. Learn how they did it in this case study.

Path

Discover how to leverage remote, flexible, on-demand testing & QA