SAP Concur Case Study

The Company

SAP Concur is an industry leader in cloud-based travel and expense-management services, offering scalable solutions designed to help employees stay focused on results—not receipts.

With an expansive portfolio of complex B2B integrations, QA is a critical part of the agile development cycle. Crowdtesting gives SAP Concur the flexibility and coverage it needs—but as it learned, not all crowds are created equal.

The Challenge

When George Andraws left Apple in 2016 to lead the QA teams at SAP Concur, he knew there would be challenges. Despite providing innovative expense-tracking solutions for today’s top firms including Uber and Airbnb, “we weren’t thinking ahead,” says George. “It was random, not well-organized and reactive … a discorded effort.”

At the time, SAP Concur was relying on pay-per-bug testing from a crowdsourced marketplace for regression testing. George says he was hopeful the model would prove motivational for the testers; however, it soon became clear that something wasn’t working.

“With our other vendor, there wasn’t a great focus on testing and quality. I tried to work with them … but we didn’t see the results we needed.”

George Andraws, Senior Manager, Quality Engineering at
SAP Concur

“They weren’t finding bugs that we were actually going to fix,” says George. “It was bug-finding, not customer-issue finding. Sure they found a lot of bugs, but a lot of them were very extreme edge-cases … there wasn’t a great focus on testing and quality.”

As the lead QA manager, George tried for months to improve the ROI, but nothing worked. “The problem,” explains George, “is that when you deal with enterprise and money and travel, things get very complicated very quickly because companies want a lot of control over the configuration. It just doesn’t work to have testers who are constantly rotating, who are more random and just follow test cases.”

“There came a point when you’d get an eye-roll every time you saw one of their bugs. It was ‘Oh no, here’s another 5 minutes of my time wasted looking into a bug we’re probably not going to fix.”

George Andraws, Senior Manager, Quality Engineering at
SAP Concur

The Testlio Difference

Concur moved on from their original vendor in 2016 and partnered with Testlio, opting for professional QA testers with dedicated, full-service teams at scale.

“Testlio’s model is different. They focus on actual testing, and finding bugs is a by-product of their great testing.”

George Andraws, Senior Manager, Quality Engineering at
SAP Concur

Since then, the results speak for themselves. Crunching the numbers reveals both total reported and resolved bugs have tripled since partnering with Testlio—and in just half the time.

“The find rate went up by 6 times. The fix rate improved. And when you look at the total number of bugs fixed.
That’s huge. That speaks volumes.”

George Andraws, Senior Manager, Quality Engineering at
SAP Concur

George credits Testlio’s dedicated-tester model, especially his point-person Rudolf whose holistic product knowledge now rivals even his own. “Rudolf has been fantastic. He knows how everything works together, so he’s able to take input from one team and see how it would impact another. He’s always thinking about quality.”

While working with pay-per-bug crowd services may offer adequate coverage, when it comes to quality and outside-the-box solutions they often fall short.

The difference was particularly apparent during one heroic effort, when a high-stakes hotfix was needed within 24 hours over a weekend. Without fail, Rudolf developed and seamlessly executed a new functional testing solution, avoiding possible service disruption and keeping customers happy.

“It really has been fantastic working with Rudolf. He is very proactive. Very thoughtful. Detail-oriented.
It’s like having someone who really cares about the quality of the product and how to test it better. That is really rare to find in a vendor.”

George Andraws, Senior Manager, Quality Engineering at
SAP Concur

QA is now an integral part of SAP Concur’s development cycle and its customer success. They are embedded in the agile and scrum teams and involved in most of the process. “We try to get involved earlier. It’s not a throw-it-over-the-wall model,” says George.

One of the biggest differences George notices with Testlio is how it helps him be more proactive, less reactive. “Testlio asks questions before we even think about them ourselves,” says George, whose teams are already one step ahead of both iOS 12 and Android P launches thanks to Rudolf.

And finally, when it comes to dynamic decision-making, it helps to have all your key data in one place. George appreciates Testlio’s comprehensive, user-focused platform and how test results integrate seamlessly with SAP Concur’s native tools of Jira + Zephyr.

“This has been an exceptional experience working with a third-party test vendor like Testlio.
There’s a personal touch, and you see that not only with Rudolf, but also with the rest of the Testlio family.”

George Andraws, Senior Manager, Quality Engineering at
SAP Concur