Breaking down the test automation pyramid

When it comes to introducing and scaling automated testing, QA teams can get quickly overwhelmed trying to figure out what processes to automate first, which tests stand to gain the most ROI, and how automation can work with manual testing strategies. Teams often look to general frameworks, like the test automation pyramid, to dive into building their automated testing strategy.

While the test automation pyramid provides a simplistic framework, its lack of context makes it challenging to implement in practice. There are hidden traps of using the test automation pyramid, one is thinking that all elements of the pyramid are implemented with the same granularity and frequency. 

Let’s break down the components of the automated testing pyramid to see where you need to really focus to succeed with this framework. The ultimate tip: Use the pyramid as a guideline to testing layers and NOT a hard and fast rule. You still need to put in the legwork to define and create your own automation framework that works for your DevOps team. 

Be bold when automating unit tests

Unit tests are your entry point, and the main focus of an immature test automation strategy. Automate these tests quickly and abundantly. Unit tests isolate and validate specific features to ensure functionality – think smoke tests, regression tests, and configuration tests. Define the parameters of “units” with your internal testing teams, since most will have varying ideas of what individual units look like. Generally, unit tests should be tests that are already broken down into specific micro-units and are relatively straightforward without any external dependencies (they are mocked or stubbed out).

Remember, automated unit tests run in isolation so they can be shifted around an automated test suite or toggled on/off as needed. Combine units to create an automated unit test suite, a collection of certified tests that assures success under normal conditions. A good test suite is stable, manageable, serves a breadth of functionality, and is fast with a clear context for developers. 

Besides the clear benefit of simplicity and high ROI, automating unit tests identifies functional issues in individual modules allowing devs to quickly fix code. Testing feedback is nearly immediate, and unit tests are often stable from run to run. After identifying unit tests to automate, developers use a test-driven development methodology to approach builds systematically.

Dev teams following the test automation pyramid strategy face a major problem with the base layer: developing automated unit tests immediately creates faster test runs, but since this pyramid section is so large, too many automated unit tests that fail can slow down the testing cycle. As your product grows and more features are added, unit tests increase, and the pyramid base gets larger and larger. A solution? Don’t test trivial code, production code, and only automate tests with clear, observable behaviors. 

Use a test methodology (like TDD)

Super simple frameworks like the test automation pyramid benefit from implementing a structure that unifies designing, building, and testing code. Here’s an example of the benefit of using a testing methodology, like test-driven development (TDD). 

When automating with the TDD methodology, devs automate existing code and only create new code if the automated code fails the test. This strategy quickens the automation process; devs are not creating new scripts for every unit test but working with existing code to automate tests before developing functional build code.

Start with the code you have. Run automated unit tests and see what passes and fails to provide results. If the code passes, great! You’ve automated your unit test. If it fails, develop new code and rerun it. 

Following TDD looks like this: 

• Write. Create a test case module with data. 
• Code. Write just enough production code to fill the test and pass.
• Refactor. If the code breaks or fails, fix the code as needed locally.

TDD is critical to automating unit tests, as it requires devs to think through a specific use case and test before building and shipping code. TDD urges developers to start with small unit tests and write simple scripts to automate (that pass!) before heading back to refactor, a key CI/CD pipeline element. 

Be intentional with integration tests 

The test automation pyramid uses “service tests,” most commonly meaning integration tests. Again, defining what exactly “integration tests” are and the boundaries of integration testing are crucial steps to getting your team automating with intent. Touch on topics like: 

• Are we testing the ENTIRE stack against other applications within the system? 
• Are we going narrow, and testing individual integration points one at a time?
• Are we only going to automate integration tests we can perform locally?

As the second layer of the pyramid, integration tests are a vital yet more complicated process to automate. Once your automated unit testing suite runs smoothly across product and dev teams, you can move toward automating unit integration tests. Don’t assume integration tests can be automated with the same frequency and speed as unit tests. For one, they take longer. There are also a variety of integration components that you need to cover. Are you running integration with specific databases? Filesystems? Internal and external APIs?

For help creating an integrated testing strategy, drop us a line.

All of that needs to be covered in test cases and test scripts. Still, with so many variables, you need a roster of talented manual testers to oversee integration and validation. Run tests parallel to each other. Once your unit tests are complete and perfected, feedback should take minutes. As that’s being run, a manual tester can start an automated integration test. QE’s can then analyze unit pass/fail and integration pass-fail. If your units are up to date and functional, unit testing will pass while waiting for integration tests to run. If both fail, manual testers can analyze flags and get results back to devs.

Separate UI tests from end-to-end processes

If you haven’t noticed, the simplicity of the test automation pyramid makes it difficult to define what each segment is truly referring to. The top of the pyramid, UI tests, are often conflated with end-to-end testing because most UI tests are done by running your app through the E2E process. 

A pyramid hack is to run UI tests without running E2E tests. Separate out individual UI and write unit tests to test your frontend code with programs like Nunit and Selenium. UI tests are harder to automate: that’s why they sit nestled at the top. For so many teams that decide UI = E2E, automation means notoriously flaky and false-positive results. Automating E2E testing workflows reach a bottleneck when false-positive results need to be manually debugged, again and again. Amidst a talent shortage if you’re team doesn’t have enough skilled quality engineers, automating end-to-end processes is costly, incredibly long, and very fragile. 

In my test automation pyramid, E2E tasks would be an even smaller tip of the iceberg; use them sparingly, and don’t expect them to provide high ROI, better speed, or be helpful. Instead, center your team on the lower blocks of the pyramid. Extend your definition of unit or integration tests, or improve your test code. By automating individual tests and taking an optimized approach, you can wield the talent to pick tasks with the highest impact potential. Like most agile processes, following the test automation pyramid encourages teams to iterate quickly and efficiently and create smarter systems, but this framework is not without pitfalls. Focus on creating and establishing definitions, boundaries, and best practices that work for your team and create better, smarter processes.