Quizlet Case Study

“As a company, we’re innovative, agile, and take our users’ experience seriously. We expect our partners to live up to that standard. The ease of communication and quality of issues reported is far better than it was with our previous vendor. Testlio helps us ship better products — I can’t imagine going back.”

Remya Ravindran, Senior Quality Assurance Engineer

Quizlet is a consumer learning unicorn that builds tools to inspire and empower students and teachers. With over 60 million monthly active users and a 4.8-star app store rating, Quizlet is among the top 20 U.S. websites and top education apps for iOS and Android.

THE CHALLENGE

Quizlet’s previous crowd testing vendor was missing priority one issues

Quizlet’s commitment to quality and collaboration has made them a leading learning and education brand. However, their previous crowd testing vendor failed to meet their high bar for finding bugs.

“We were finding too many critical issues that they had missed,” said Remya Ravindran, Senior QA Engineer. “Their testers kept changing and didn’t understand our product. The ‘bugs’ they reported were minor, often UI related, instead of functional P1 issues.”

Inconsistent communication and transparency challenges with the previous vendor also created bottlenecks in Quizlet’s software delivery process. Remya had to take extra time to write manual test steps and deal with last-minute questions within tight weekly release windows.

With 60 million passionate users, missed issues had the potential to interrupt important study sessions for students. To ensure Quizlet addressed these issues as soon as possible, the user operations team would share them in a wider Slack channel, making the broader team aware of the issues that were not caught.

Remya did like the crowd testing model, seeing the value of unbiased QA and flexible weekend testing, but escaped issues and a lack of transparency were deal breakers. “We’re innovative, agile, and take our users’ experience seriously. We expect partners to live up to that standard.”

THE Solution

Quizlet brought in Testlio as an extension of their internal team

The Quizlet team started their search for a new software testing partner that could meet their rigorous internal standards, catch issues before they reached customers, and keep their quality team out of the spotlight of the internal incident report channel.

“I wanted more transparency and higher quality issues reported – a vendor that would feel like an extension of our internal team,” Remya recalled.

With expert support from an industry-leading engagement team and a dedicated group of carefully vetted freelance testers, Testlio’s networked testing approach stood out from their legacy crowd testing vendor.

Testlio and Quizlet established a cadence of regular check-ins as well as quarterly business reviews to align and iterate on the overall testing strategy by discussing challenges, learnings, and upcoming company initiatives. From the start, Testlio became a true partner to Quizlet’s internal quality team.

“My confidence in Testlio has grown so much to where their team has access to JIRA tickets for the projects they work on,” said Remya. “They tag my engineers in our internal Slack when they have questions for them. With our previous vendor, we didn’t know the testers, so this wasn’t possible.”

THE Outcomes

Quizlet finds more P1 issues and releases on time with confidence

Today, Quizlet releases confidently each week and benefits from new efficiencies in their testing process. Instead of holding them back, their crowd testing partnership with Testlio allows them to do more of what has made them successful – rapidly shipping new products and updates that help their users learn and grow.

Testlio relieves release day stress by catching bugs before they light up the incident report channel. Remya estimates that Testlio’s expert testers find 25% more priority one issues than their previous vendor and catch 20 bugs each year that would have led to crashes.

“The quality of testers is great, and the quality of the bugs reflects that,” she said. “They’re also not changing all the time, so they know the product. I don’t spend energy explaining things or spoonfeeding them.”

Speed has also increased as testing bottlenecks disappeared. In two years of partnership, Quizlet has never delayed a release due to problems with Testlio.

“We release weekly now, so speed is very important,” said Remya. “We send builds on Friday, then Testlio works overnight and over the weekend to test and file detailed reports. By Monday morning we know what is wrong and if there are any blockers or showstoppers that need to be fixed. The rollout is everything and I know that we’ll release on time with Testlio.”

The transparency, consistency, and direct communication with Testlio’s team have significantly improved engineering efficiency as well. Because Testlio has access to JIRA tickets for the releases they are testing, she no longer needs to create test steps manually or answer last-second questions on Fridays. Instead, the Quizlet team can focus on delivering quality software and managing smooth weekly releases.

“By the time we release, we are confident that anything the end-user would have caught has been found by Testlio, preventing a huge percentage of would-be crashes. The bar for our partnership is very high and continues to grow stronger.”

Remya Ravindran, Senior Quality Assurance Engineer