Unlocking QA Coverage Efficiency Through Signal-Driven Testing

Emeka Obianwu , VP, Alliances and Acquisitions

November 8th, 2023

Signal-Driven Testing (SDT) is a core component of Fused Software Testing, a transformative framework and approach to software quality assurance. SDT focuses on the opportunity for product teams to make better and faster product quality decisions by aggregating, synthesizing, (signaling) and actioning data that exists across internal and external systems. SDT represents an acknowledgement that data can and should play a more strategic role towards helping product teams optimize the coverage and cost of software quality assurance.  

In this guide, we’ll define SDT, offer examples of its implementation, and highlight the strategic benefits to product teams. 

What is Signal-Driven Testing?

To bring clarity to the definition of SDT, it helps to establish one upfront distinction. So let’s start with what is not Signal-Driven Testing. The widely adopted set of tools on the market that are utilized to manage the “work” of software testing. These systems offer a range of capabilities including serving as a system of record for test cases and facilitating collaboration around issue management. These systems also incorporate business intelligence (aka reporting) capabilities to help product teams gain more comprehensive visibility into both the quality of the product and the scope of the testing conducted. These capabilities help answer important questions like test coverage, defect trends, and product quality over time. SDT should not be compared to these tools. SDT is not in the business of capturing and organizing data to report on the work and outcomes of software testing. In fact, SDT is not about reporting at all. 

So, what is Signal-Driven Testing?

SDT broadens the scope of systems that can contribute data to make software testing smarter, more responsive, more impactful to the business. This does include the aforementioned test management systems, but goes far beyond. It includes the entire DevOps toolchain utilized by product teams to build, test, manage, and measure software development. It goes even further, incorporating 3rd party sources of data to the degree those data sources can offer actionable QA insights. 

Although SDT broadens the scope of data to inform software testing, it looks at that data through a very focused lens. The lens is defined by signals. Signals are automated notifications that are actionable, contextual, and real-time or near real-time. Signals are generated by rules that are applied to data from across one or more systems. SDT signals focus specifically on helping product teams direct and redirect QA capacity:

• Highlight product quality “hot spots” that require more QA coverage

• Trigger and guide human action against automated test run results

• Identify product quality issues (bugs) that have “escaped” into production

Integrations Power Signal-Driven Testing

SDT is by definition tool agnostic, relying on an open architecture to tap into varying systems for the data required to generate signals that optimize software testing. While product teams should continuously seek to identify and unlock new sources of data, the following systems should be considered for an SDT data architecture:

Benefits of Signal-Driven Testing

In the modern era of ubiquitous digital transformation, brands compete on the basis of delivering continuously compelling mobile and web experiences. Speed of delivery is a competitive advantage. But release speed cannot come at the expense of the end user experience. As a result, all product teams face the potentially high stakes balancing act between increasing software delivery speed and ensuring product quality meets end user expectations. Finite QA capacity means product teams must continuously seek to align those precious resources with the most important quality assurance activities. SDT unlocks the added intelligence that product teams need to more optimally deploy finite QA capacity. The more optimally QA capacity is deployed, the less likely quality becomes a bottleneck to software delivery speed. SDT is about enabling product teams to continuously optimize test coverage, allocate resources more efficiently, and shorten time-to-resolution on product quality issues. 

• Agile Test Coverage: Unfortunately many product teams suffer from static test coverage, limited by a set of documented manual test cases or automation scripts. Much as software development has moved towards iterative delivery models (continuous delivery, agile, etc.) so should QA. SDT fosters increased QA agility with proactive signals that continuously inform where test coverage should be redirected to keep pace with product innovation and end user feedback loops.
• Intelligent QA Staffing: Software engineering teams employ a range of skill sets across the SDLC to achieve testing coverage. Engineers adopt “shift left” testing practices, QA analysts perform manual testing associated with scheduled releases and on-demand ad hoc validation, and quality engineers are tasked with automating manual testing for increased speed and coverage. SDT increases the combined efficiencies across staffing by offering intelligence (signals) to both inform when these skills should be applied and to streamline the handoffs between the teams. 
• OpEx Efficient Test Coverage: SDT implements a more data-driven approach to achieving QA coverage. It acknowledges that absent infinite resources, no engineering team can test every single user, language, device, configuration and location combination. So QA is inherently about optimizing how a finite set of resources are applied to achieve coverage. It seeks to generate signals to direct resources where they are going to have the most significant impact (mitigate risk) on product quality.
• Customer-Centricity: Product teams already understand the value of seeking end user feedback to inform application quality. Tools like feature flagging are now pervasively utilized to gain end user insights into quality and usability. SDT embraces such tools but takes a more holistic end user data strategy (feature flag insights, customer support tickets, app reviews…) from which signals can be generated.

Launching Signal-Driven Testing

Creating a SDT strategy and architecture requires systematically (over time) activating a set of signals into engineering and QA processes. As with many transformational initiatives that both augment and change existing processes, it is best to start small and build on early success. Consider the following approach to embark on the SDT journey: 

1. Signal Definition: Identify a prospective signal that will have a meaningful impact on directing or redirecting QA capacity. Focus on leveraging data already accessible via existing integrated systems. 
2. Validate Business Process Viability: How would the business process need to change when the signal is activated as part of the process? Make sure the process lends itself to actioning the signal.
3. Data & Rules: Enable filters and rules against the data to ensure signals are automated, meaningful, and actionable by internal teams. Consider GenAI as a prospective tool to interpret or summarize data sets.
4. Measure Impact: The measuring stick for SDT is getting more test coverage from existing or less QA capacity. How is test coverage improving or more optimized as a result of the signal-powered business process?

Examples of Signal-Driven Testing in Action

Discover the Power of Signals Today

The modern era of software development demands more from product teams. Keeping pace with evolving expectations for transformative digital experiences, fending off disruptive new industry entrants, and keeping budgets in check can be collectively a tricky proposition. It’s a dynamic that challenges product teams to become even more data-driven across engineering practices. Signal-Driven Testing offers these teams a pathway to a more diagnostic approach to quality assurance. SDT transforms QA from a set of workloads applied across the SDLC to a capacity model that can be dynamically applied over time based on signals that inform where resources are most needed to ensure high quality products. Have questions about this emerging methodology? Reach out to Testlio’s experts today.