Trip Report: On The Road to Signal-Driven Testing

, VP, Alliances and Acquisitions

Just shy of a year ago and coinciding with the Atlassian Team ‘23 conference, Testlio unveiled an initiative to help product teams adopt signal-driven testing as a core pillar of the future of software quality engineering. A lot of exciting things have happened and continue to happen since that announcement, which collectively serves as validation of the opportunity for product teams to dramatically improve test coverage efficiency through signal-driven testing.

First, a Quick Refresher on Signal-Driven Testing

  • SDT is an approach to quality engineering that generates data-driven quality insights (signals) to speed time-to-resolution on issues or uncover gaps in test coverage
  • Signals (in the context of SDT) are generated by integrating and correlating data (filters, rules, GenAI) across systems: the DevOps toolchain, support systems, and user sentiment forums.

Sights and Sounds on the Journey

Let’s start by highlighting two key observations from the last year that I think help bring directional clarity to the SDT roadmap.

First, SDT is not test automation. Introducing the concept of SDT sometimes leads to the question: isn’t SDT just a flavor or evolution of test automation? Equating the two disciplines does a big disservice to test automation and artificially limits the scope of SDT use cases. Test automation leverages machines and technology (e.g. RPA, visual AI) to replace manual testing so product teams can keep pace with increased demand for software delivery speed at a manageable cost. Test automation is a mainstay of the modern DevOps toolchain, integrated with the continuous delivery pipeline. SDT doesn’t seek to automate testing workloads but instead makes those testing workloads smarter. Test automation is both a potential benefactor of SDT (receiving signals that inform test automation workloads) and a potential contributor (helping to generate signals that inform testing workloads downstream of test automation). 

Second, GenAI will be a force multiplier for SDT. We are in the era of GenAI, and quality engineering is one of a long list of business processes that will see significant and lasting disruption. 

  • Changing how we test software by making the process better, faster, cheaper
  • Changing what we test to ensure GenAI-powered user experiences are safe and high quality 

Testlio previously incorporated GenAI into our delivery platform, and we have now seen two-thirds of our clients’ workspaces in the platform leverage GenAI to improve the quality and speed of both test case management and issue management. The next chapter is applying GenAI to SDT, initially in the context of user sentiment signals. This will include leveraging GenAI across user reviews, social forums, and support tickets to speed up the time-to-resolution of escaped issues and identify test coverage gaps. 

So, where are we on the SDT journey? We are still early, but as signaled (pun intended) earlier, there has been strong validation from the early adopters of SDT. It’s most effective to provide validation through examples of enterprise product teams benefiting from their journey to SDT.

Example of the Potential Impact of SDT

Let’s walk through an example where a company, let’s call it Acme Co, is leveraging Testlio’s SDT integrations and approach.

Under an active managed services testing engagement, the Signals module of the Testlio Platform has been activated across Acme Co’s workspaces. Testlio’s dedicated client services team worked with Acme’s product team to configure a set of rules in the Testlio Platform to generate signals based on user reviews across the Google Play and Apple App Stores. These rules are tailored to the specifics of Acme’s industry and application experience. On a daily basis signals are generated in Acme’s workspaces that trigger notifications for the Testlio client services team. On behalf of Acme, our team proactively triages these signals. A recently generated signal uncovered a configuration-specific issue on the Android platform that rendered embedded search inoperable under certain conditions. The signal led to both faster time-to-resolution on an escaped issue and updates to the test suite to ensure future coverage.

While the above scenario offers strong validation for SDT, it is just one of many prospective use cases. Other client engagements are tackling social forum signals (eg. data from Reddit, X/Twitter, etc.) and fused testing signals (data from test automation and observability products). The future is bright for SDT, and product leaders will increasingly challenge their teams to extract hidden value from systems data to get a better ROI (speed, coverage, quality) on software testing.