“As a company, we’re innovative, agile, and take our users’ experience seriously. We expect our partners to live up to that standard. The ease of communication and quality of issues reported is far better than it was with our previous vendor. Testlio helps us ship better products — I can’t imagine going back.”Remya Ravindran, Senior Quality Assurance Engineer
Quizlet’s previous crowd testing vendor was missing priority one issues
Learning & Education
Web, iOS, Android
Quizlet’s commitment to quality and collaboration has made them a leading learning and education brand. However, their previous crowd testing vendor failed to meet their high bar for finding bugs.
“We were finding too many critical issues that they had missed,” said Remya Ravindran, Senior QA Engineer. “Their testers kept changing and didn’t understand our product. The ‘bugs’ they reported were minor, often UI related, instead of functional P1 issues.”
Inconsistent communication and transparency challenges with the previous vendor also created bottlenecks in Quizlet’s software delivery process. Remya had to take extra time to write manual test steps and deal with last-minute questions within tight weekly release windows.
With 60 million passionate users, missed issues had the potential to interrupt important study sessions for students. To ensure Quizlet addressed these issues as soon as possible, the user operations team would share them in a wider Slack channel, making the broader team aware of the issues that were not caught.
Quizlet brought in Testlio as an extension of their internal team
49 expert testers
The Quizlet team started their search for a new software testing partner that could meet their rigorous internal standards, catch issues before they reached customers, and keep their quality team out of the spotlight of the internal incident report channel.
“I wanted more transparency and higher quality issues reported – a vendor that would feel like an extension of our internal team,” Remya recalled.
With expert support from an industry-leading engagement team and a dedicated group of carefully vetted freelance testers, Testlio’s networked testing approach stood out from their legacy crowd testing vendor.
Testlio and Quizlet established a cadence of regular check-ins as well as quarterly business reviews to align and iterate on the overall testing strategy by discussing challenges, learnings, and upcoming company initiatives. From the start, Testlio became a true partner to Quizlet’s internal quality team.
Quizlet finds more P1 issues and releases on time with confidence
Today, Quizlet releases confidently each week and benefits from new efficiencies in their testing process. Instead of holding them back, their crowd testing partnership with Testlio allows them to do more of what has made them successful – rapidly shipping new products and updates that help their users learn and grow.
Testlio relieves release day stress by catching bugs before they light up the incident report channel. Remya estimates that Testlio’s expert testers find 25% more priority one issues than their previous vendor and catch 20 bugs each year that would have led to crashes.
“The quality of testers is great, and the quality of the bugs reflects that,” she said. “They’re also not changing all the time, so they know the product. I don’t spend energy explaining things or spoonfeeding them.”
Speed has also increased as testing bottlenecks disappeared. In two years of partnership, Quizlet has never delayed a release due to problems with Testlio.
“We release weekly now, so speed is very important,” said Remya. “We send builds on Friday, then Testlio works overnight and over the weekend to test and file detailed reports. By Monday morning we know what is wrong and if there are any blockers or showstoppers that need to be fixed. The rollout is everything and I know that we’ll release on time with Testlio.”
“By the time we release, we are confident that anything the end-user would have caught has been found by Testlio, preventing a huge percentage of would-be crashes. The bar for our partnership is very high and continues to grow stronger.”Remya Ravindran, Senior Quality Assurance Engineer