Testlio Experiments Show Generative AI Can Help Experts

Tim Ryan , Tim Ryan serves as the Director of Marketing for Testlio and spends his time between Austin, TX and New Orleans, LA.

May 25th, 2023

New quality assurance paradigm helps companies efficiently release great digital products

May 25, 2023, Austin, TX — Testlio, a leader in technology-powered software testing services, announced today that for 30 days across April and May 2023, it ran two experiments using generative AI technologies. These experiments signaled that quality assurance can be better and faster using a combination of expert quality professionals and AI-powered machines. 

With these findings, Testlio has embraced commercial-grade generative AI capabilities via its software testing platform and services. It will use new efficiencies to help its clients—including some of the world’s largest digital product companies—release better products faster.

Testlio’s first two experiments validated core hypotheses about the application of generative AI to quality assurance for digital software products:

• Generative AI can help humans work more quickly. Testlio’s initial experiments showed that skilled testing managers can refactor test cases 15-30% faster with generative AI-assisted tools versus traditional techniques.
• Generative AI can help humans produce better results. Testlio’s initial experiments showed that issues (i.e., bug reports) created by skilled quality assurance testers leveraging generative AI tools had 40%+ fewer errors than issues created using other software tools.

The impact of these findings is significant for the software testing industry. Analysts estimate that software testing costs companies ~$45B per year¹, with spending expected to grow 5% annually. With new generative AI-powered approaches, companies can expand their software testing coverage and gain even more confidence in the quality of their product releases. 

“Artificial intelligence is both disruptive and invigorating to the world of software testing,” said Steve Semelsberger, CEO of Testlio. “Our initial experiments showed that generative AI can help quality assurance professionals produce better results. We are optimistic about using generative AI to help our clients ship better digital products faster.”

Broadly, Testlio experiments show that generative AI can make written instructions and information more understandable, leading to more efficient testing and better quality assurance results. Generative AI can reduce time spent writing, reviewing, and editing core instruction content, which is critical to successful quality assurance and quality engineering efforts. 

Generative AI can also help reduce errors. For example, one experiment showed that grammatical errors decreased from an average of 1.3 per issue to 0.7 (a 46% reduction).

The recent Testlio experiments leveraged commercial-grade generative AI systems via licensed OpenAI corporate offerings. Artificial intelligence capabilities were inserted into key workflows in the Testlio software testing platform. Scrubbed and protected data sets from sample client environments were used to run tests using generative AI-assisted techniques versus control processes (standard approaches). These processes were monitored closely via a working group led by senior quality expert Rudolf Elbrecht in partnership with chief technology officer Jaanus Ranniit and manager of security & DPO Matten Mätlik. 

“With carefully executed real-life experiments, we were able to show that generative AI makes critical software testing processes faster and better,” stated Rudolf Elbrecht, Sr. Director, Quality Assurance Center of Excellence, Testlio. “However, the key to success is combining skilled specialists with generative AI tools.”

Testlio is in the process of rolling out experimentally validated generative AI services to its clients. It will initially use Microsoft Azure AI (powered by OpenAI). It also intends to leverage Google PaLM and Amazon Bedrock services, ensuring the best models are deployed for critical software testing workflows. It will also continue to run experiments to better understand additional applications of artificial intelligence tools to software testing.

Please note: Testlio has performed security, privacy, data, and legal reviews of Microsoft, Google, and Amazon generative AI offerings through the lens of quality assurance. It believes that these companies each offer strong frameworks to ensure Testlio client data is safe and protected. As it rolls out generative AI services for clients, Testlio will also partner directly with large customers that have specific data protection terms, ensuring that generative AI approaches are compliant. 

1. Source: Global Market Insights, March 2023

About Testlio:

Testlio is a software testing company. We are the originator of fused software testing, a unique approach that combines expert humans and powerful machines to help digital innovators deliver quality products at scale. In any location. On any device. In any language. Via any payment instrument. The company is distributed by design, with full-time people in 30+ countries and flexible freelancers in 150+ countries. Clients include Amazon, athenahealth, Fox, Microsoft, the NBA, PayPal, Uber, Wayfair, and many more. Collectively, they have awarded us an industry-leading 4.7 G2 rating. To learn more, visit www.testlio.com.