Modern engineering and product teams must meet an increasingly high bar for quality and speed. While QA automation is marketed as a quick way to increase cadence to meet consumer demands, fusing manual and automated strategies creates a more effective, failsafe approach. Automated testing ensures consistency and immediate test execution, while manual testing offers flexibility and actionable insights. 

Yes, automation has immense benefits: accelerating repeatable tests, broadening coverage, critical flexibility, and cost savings. But automation is not a one-step process. You must first invest in staff + external tools to manage automated testing, identify candidates for automation, and have a continuation plan for flakey scripts.

Jumbotron image

Step 1: Perfect your manual testing

Ok… perfect is a strong word. But the first step to assessing if you’re ready for automated testing is to ensure your manual testing is efficient and effective. 

Hastily writing scripts to automate test cycles and get back to development is tempting, but automating bugs is a major issue. And for QE’s who are converting manual test cases into automated scripts, it’s tempting to write the entire end-to-end manual test case into code. Unfortunately, both approaches lead to messy test suites and more maintenance time in the long run. Following software development principles is vital when writing automated test scripts, such as source code management, using design patterns, and investing in frameworks.

Manual testing –> Automation
The initial testing focus for Testlio’s NFT marketplace client was to leverage on-demand manual testers to check the user lifecycle, like sign-ups, purchases, and small interactions within their platform. Over time, as the input and output parameters of the manual tests became repeatable, much of the testing became automated, leading to a 98% reduction in time to test. 

  • Have plentiful manual test cases that are easily repeatable, efficient, and effective. 
  • Consider software development principles when writing automated test scripts and developing an automated QA strategy.

Step 2: Know what to automate and what needs a human touch

Where manual testing excels: 

  • As a precursor: If you try to automate before starting or perfecting manual tests, you could automate bugs into your automation.
  • Feature testing allows testers to independently test a new feature until it’s relatively bug-free without compromising the functionality of the entire project.
  • UI and UX testing
  • Failed tests. If an automated test fails, it’s usually up to a human to judge whether the test or the product code is at fault.
  • Exploratory testing for higher-level assessment of complex business flow and real-life situations such as interruptions and display image orientation. 

Where automation excels:

  • Unit tests. Start with unit tests to ensure that an automated testing suite is broken down into small parts, reducing your automated testing operation’s flakiness and run time. 
  • API tests run pretty fast, but not as quickly as unit tests. They are also reliable because they often run without providing false results/ flaky tests. 
  • Supporting DevOps to improve results velocity and provide development teams fast feedback.
  • Repetitive and data-intensive tests.
  • Happy path testing using known inputs and a clearly defined expected output.

While building your QA automation strategy, be mindful of time. It may take weeks to months to develop, write, and build a full test automation suite. In the meantime, manual testers can serve as backup for overnight and weekend regression testing while you work to build your automated suite. 

  • Determine candidates for automation.
  • Determine what tests should remain in human hands (and when you need manual fallback).
  • Create a roadmap for automation using the test automation pyramid.

Step 3: Stack internal + external talent

To tackle QA automation, software companies need dedicated and knowledgeable Quality Engineers. Easier said than hired. A major hiccup on the road towards automation is the scarcity of qualified tech talent and the cost of recruiting and maintaining a full-time staff. 

Before executing your automation strategy, take stock of your internal talent. Identify holes for improvement and where you need to invest time and resources to take on automation successfully. Which employees have the skillset to write, implement and maintain scripts? Who can interpret the signals created by automated testing platforms? Who can build workflows and connect integrations? 

Heading to external partners and their integrated tools is vital to address capacity needs. For example, Testlio’s freelance network includes on-demand QEs to create, run, and optimize automated test scripts. They leverage frameworks like Appium and Selenium and help integrate with test automation platforms like Applitools, BrowserStack, Headspin, mabl, Sauce Labs, and Suitest. 

  • Have a mix of QA and QE resources to oversee automation script creation, execution, and maintenance. 
  • Create plans to translate and action the signals/results being generated by test automation tools.

Step 4: Tackle third-party tools, frameworks, and integrations

Now that you have a handle on your internal talent and have manual QA and QEs at your disposal, step four is to become familiar with automated testing tools and how they work. 

Spend time building a kickass tech stack that helps handle automated test creation, execution, feedback, and test runs. Use partners to maximize your investment in automated testing capabilities.

As part of your CI/CD pipeline or Agile planning processes, create a game plan for how to best stack and utilize these tools. Which tool are you using for what feature? How will you integrate signals from each into a more effective strategy? Embrace new partnerships to keep up with frequent release cycles, maintain a staff of expert software testers, and address capacity needs. 

Questions for a potential partner.

What types of QE and QA resources do they offer? 

How do they plan to manage your automated test suite?

What are their approaches to DevOps integrations? Do they have existing partners?

Are they similarly equipped with manual and automated testing strategies to maximize coverage across humans and machines?

  • Make test automation investments in tools like Applitools, BrowserStack, Headspin, mabl, and Sauce Labs.
  • Plan to integrate CI/CD, automated testing, and task management tools with a manual testing platform.

Step 5: Plan for automation failure

The final step in this checklist is planning for the inevitable hiccups that occur with QA automation. Automation is notorious for coming to a screeching halt in the presence of flakey scripts. Implement fast-follow workflows where expert manual testers validate the results of automated scripts.

Take an incremental approach to automation. For example, before your first automated run,

it can take weeks to build a framework, design tests, write the code, and “test the tests.” The risk is that the app may have changed too much by running your first automation.

Start with a single test to automate. Then, incrementally add to that test, keeping the test suites running daily and updated alongside the app as it’s updated. Adapt frameworks like test-driven development (TDD) to your automation strategy.

Finally, in the presence of these failures, be ready to tinker with your strategy. If automated runs continuously produce noisy or flaky results, convert them to manual runs until you can rework the scripts. Conversely, if manual testers run the same repetitive tests with clear results, shuffle their runs into the “to be automated” pile.  

  • Implement fast-follow workflows where expert manual testers validate the results of automated scripts.
  • Connect and adapt existing test automation initiatives (eg. shift left, shift right, continuous development) to reap higher ROI.
  • Convert flaky automated runs into manual and repetitive manual runs into automated.

Kassidy Kelley serves as the Managing Editor for Testlio and works from her home base in Boston, MA.