A Product Leader’s Guide to Automated Testing: Effective Strategies and Potential Pitfalls

, Director of Quality Engineering

Automated testing has emerged as a prominent practice that leverages machines (compute power, scripts, machine learning, genAI) to validate the quality of a software application.

With the promise of increased efficiency, broadened test coverage, and accelerated software delivery, automated testing has become a cornerstone initiative for modern software engineering teams. In this guide, we explore the role of automated testing from a product leader’s perspective, its potential game changing benefits, and the practices that are critical to long term success.

The Ingredients for Effective Automated Testing

Developing an effective automated testing strategy requires a comprehensive and sequential approach.

The following practical guidelines will help product teams avoid common pitfalls, maximize the benefits of test automation, and ensure the program is built-to-scale.

  • Align on the Roadmap for Automation: Not all test cases are suitable for automation. Some test cases are uniquely suitable for human insight (manual) scripted functional or exploratory testing. Also start by selecting test cases associated with more stable features of the application. These are ideal candidates to ensure a good ROI on early automation efforts because they avoid the higher total-cost-of-ownership associated with automation that must keep pace with high rate of change features. 
  • Establish a Unified Automation Framework: There is no shortage of automation frameworks available in the market today but selecting the one, most suitable to address automation across your supported platforms and operating systems, requires careful consideration. The implications of this decision will be long lasting and highly impactful to the long term ROI on automation because of the inherent tradeoffs across frameworks such as supported scripting languages and portability.
  • Define a Governance Process: These are the internal set of processes and systems that will serve as ongoing checks and balances to ensure automation is consistently developed and implemented the right way. If you think of the automation framework as the materials that will be used to build the house, the governance process is what ensures the builders consistently use the materials in the right way to construct the house. Examples of governance include reusable libraries, runbooks, pull request processes and automation script peer reviews.
  • Testing Infrastructure and Tools Selection: Establishing a technology tools architecture for automated testing is a critical element of the overall strategy. Note this doesn’t mean acquiring or deploying the tools that make up the architecture on day one. It does mean recognizing the different types (functional, visual, low code, public device farms, etc.) of automation technology and the role they play towards fulfilling the automation roadmap over time. Engineering teams may need to combine multiple tools or runtime environments to fulfill the long-term automation roadmap.

The Interplay Between Automated and Manual Testing

Automated testing is a highly strategic lever for engineering teams pursuing greater efficiencies on test coverage and it will only become more impactful in the era of machine learning and GenAI. But there is no panacea in which 100% of test cases are automated. Engineering teams that build a strategy and resulting roadmap with this mindset are doomed to achieve a lower ROI on automation.

Instead teams should seek to avoid a zero sum mindset when it comes to the relationship between manual and automated testing. Establish upfront a set of test cases for which manual testing is uniquely suited, for which human insights offer the brand a competitive advantage in the market.

Update this list over time to reflect the evolution of the application flows, functionality, and the shifting competitive market dynamics. Additionally identify compelling opportunities to augment automation and shift left testing with manual shift right testing. The interplay between the two testing types offer powerful opportunities for both speed to market on application releases and better test coverage. 

Examples of fusing manual and automated testing:

  • Manual Fallback. Automation run failures trigger burst into manual bug verification or validate location and device specific issues.
  • Feature Flag Crowdtesting. On-demand access to crowdsourced manual testers for faster feedback loop on feature beta and A/B testing.
  • Signal-Driven Automation. Leverage manual testing to inform updates to automation test scripts for improved coverage, script reliability and reduced false positives.
  • Automation Assisted Manual Testing. Leverage automation to conduct build validation and smoke testing to increase the speed and impact of manual testing.

Test Automation Speed Bumps

While test automation is potentially game-changing to software testing, a common set of challenges have undermined adoption and return-on-investment. Product teams should establish a strategy upfront for test automation that acknowledges and addresses these challenges.

  • Unsustainable Access to Automation Talent: Finding and retaining the quality engineering talent required to engineer, maintain, and triage results associated with a test automation suite can be challenging.  Engineering leaders should develop a plan for staffing that places a premium on flexibility over time. The inherent nature of the test automation journey requires a staffing model that can adapt over time (increase and decrease) to meet the evolving talent type and capacity requirements.
  • Software Design and Quality Engineering Misalignment. Test automation should be considered a core pillar of software design practices. Absent this approach there will inevitably be a divergence whereby software design practices start to undermine automation. Architecture decisions made by developers can have a lasting impact on the feasibility and TCO of downstream automation efforts. 
  • Missing the Mark on Device Coverage: Automated testing doesn’t remove the requirement to test across the spectrum of device types and configurations that are reflective of the application’s user community. The testing infrastructure available for automation should include runtime environments that deliver comprehensive real device coverage. The combination of a device farm and access to local devices for automation runs can give product teams a cost effective way to achieve broad device coverage for automation.
  • Technology Lock-In Limits Adaptability. There is a lot of promise across the test automation technology and tooling landscape, but product teams must remain vigilant about the selection process and the implementation configurations. Both the IP ownership and portability of automation must be preserved and therefore should be a top criteria when selecting and implementing automation technologies.
  • Approaching Automation as a Solution Not a System. A lot of automation initiatives have failed because product teams adopted automation as a point-in-time investment to increase velocity and increase engineering capacity. The inherent ongoing maintenance and end-to-end automation run management processes were underinvested or ignored altogether. Investing in automation the right way requires a pragmatic approach that might require a slower pace to ensure those investments are built to scale the ROI over time.

More Resources for Automation Testing

Now that you’re up to speed, check out these resources to gain even more insight into automated testing:

Whether you’re looking to implement new automation strategies or improve existing ones, Testlio provides the expertise and tools necessary to achieve efficient and effective quality assurance. Visit our Test Automation Solutions page to learn more about how we can speed up your testing efforts! 

Discover how Testlio can help you maximize the potential of your test automation.