eCommerce Payments Testing Case Study

The Company

Our client is a publicly-traded e-commerce company with several global offices and 40+ million active customers. Ranked in the top-15 U.S. iOS shopping apps, they have one of the highest Apple App Store ratings in the segment. They are also within the top-20 shopping apps on the Google Play store.

After a successful pilot, this e-commerce leader selected us to provide managed application testing for their global expansion. Specifically, Testlio tested their web app through a complicated 10-country buildout using expert testers for payments testing, localization testing, user experience testing, and more.

Today, Testlio’s distributed and experienced testers flex in and out of assignments, helping with launches, market updates, significant events, and more.

The Challenge

Offering a world-class customer experience is built into how our client develops, tests, and releases new versions of their app. Their rapid expansion into new global markets required in-country expertise to evaluate the customer and merchant purchase experience.

Their Challenge? A highly skilled in-house testing team lacked the global network required for localization and payments testing of different currencies, banking systems, symbols, languages, fees, shipping services, and more.

The client needed a fully managed yet economically viable testing service to augment their in-house QA team, vet and manage testers, and deliver results. Specifically, they needed payment testing experts with real devices in 10 target countries, playing either the role of a shop owner or buyer.

Their payment testing challenges:

• Assess financial transaction functionality and learn how to simplify the user experience for purchase, shipping, and refund
• Test to assure the disbursement of funds to and from in-country bank accounts
• Testing the new customer and new shop owner onboarding experience by country
• Selection of experienced and trustworthy testers willing to use their bank accounts to play the role of both customer and shop owner
• A flexible testing partner to manage testing and reporting and ensure rapid turnaround aligned with the schedules of the US-based engineering team

The Solution

Expanding retail and commerce brands struggle to find an efficient way to test their apps in other countries. In our client’s case, they needed to quickly identify and fix issues to ensure a consistent user experience in countries across the globe.

Testlio tackles the localization testing challenge with powerful test management software, services, and a global network of expert testers available on-demand exactly when and where needed.

Where to start?

To kick things off, the client opted for one of Testlio’s European services teams to provide day-to-day management responsibilities, filter test results, and ensure on-time delivery of reports. The Testlio team also hand-selected expert in-country testers from a pool of over 10,000. The services team provided daily reports with screenshots, questions, project status, context information, and recommendations. The reports arrived by 7 a.m. EST so the client’s U.S. development team could fix issues immediately.

The flexible testing team came from 10 countries across four continents and conducted several testing tasks per day covering essential device/OS combinations. The testing types included functional, payments, exploratory, and remote user experience tests. The on-demand testers were compensated hourly, at above-market rates aligned with the cost of living for each country.

This compensation model differs from the traditional bug bounty model and is unique to Testlio. Offering competitive hourly rates builds two-way trust by rewarding expert testers to collaborate, stick with a client project over time, and provide thoughtful insights and issue identification. Speaking of trust, the testers used funds from their personal bank accounts. This mandate ensured a real-world app-driven buying and selling experience where testers evaluate usability and speed of transaction from every angle and in every market.

Let the tests begin

Working in short 1-1.5-hour bursts and agile testing windows, the global testing team integrated seamlessly with the U.S.-based development team – working across timezones to test overnight and delivering issue reports to the U.S. engineering team as they began their workday. Testlio’s test management software streamlined the entire testing experience by managing task reporting and test cycle invites.

Outcomes

The client ranks in the top five of eCommerce Bytes’ 2020 Sellers Choice Marketplace Ratings, the Top 8 Best eCommerce Marketplaces for Online Sellers by Sellbrite, and a Top 10 Marketplaces to Sell Online by ASD Market Week.

Their rating on the App Store is a world-class 4.9 stars, and they’ve earned the title as one of Fast Company’s ‘10 Most Innovative Retailers for 2020.’

There were many specific examples of issues uncovered during testing. In one case, testers intentionally simulated a typo at the point of purchase. The transaction didn’t go through – but there was a bug that caused a delay in the confirmation notification. The client used this information to modify their customer notification tools.

In another case, testers discovered a value misinterpretation of their country’s penniless currency at checkout. A bug caused purchases without an even amount to round up, in some cases doubling the cost of an item (e.g., $1.05 became $2). The client used this information to develop a localized payment system and pricing model to prevent price round ups.

Another outcome came about from testing a series of payment channels for functionality. The testers discovered that varying bank account number formats prevented one of the world’s largest payment platforms from facilitating purchases. Fixing this issue helped prevent future blocked purchases.

What started in 2017 as a 100-hour web application testing pilot has expanded into a long-term partnership where Testlio’s distributed and experienced testers flex in and out of assignments, helping with launches, market updates, significant events, and more.

With a highly selective testing vetting process, client NPS score of 71, and a 4.8/5 star rating on G2, Testlio provides an industry-leading approach for clients’ holistic web and mobile app testing needs.

With a highly selective testing vetting process, client NPS score of 71, and a 4.8/5 star rating on G2, Testlio provides an industry-leading approach for clients’ holistic web and mobile app testing needs.