Testing pyramid: Breaking down the layers Kassidy Kelley , Managing Editor February 28th, 2022 When it comes to introducing and scaling automated testing, QA teams can get quickly overwhelmed trying to figure out what processes to automate first, which tests stand to gain the most ROI, and how automation can work with manual testing strategies. Teams often look to general frameworks, like the test automation pyramid, to dive into building their automated testing strategy. What is the test automation pyramid? The Testing Pyramid is a framework designed to help both developers and QA teams create high-quality software. It helps developers quickly identify if a change they’ve introduced has broken the code, contributing to a more reliable test suite. The testing pyramid, also known as the test automation pyramid, outlines the types of tests that should be included in an automated test suite, as well as the sequence and frequency of these tests. The primary goal is to provide immediate feedback, ensuring that code changes do not disrupt existing features. While the test automation pyramid offers a simple framework, its lack of context can make practical implementation challenging. One common pitfall is the assumption that all elements of the pyramid should be implemented with the same granularity and frequency, which is not the case. The test automation period works in three layers. Unit Tests – At the top of the pyramid Integration Tests – The middle layer End-to-End Tests – The bottom, largest layer of the pyramid Let’s break down the components of the automated testing pyramid to see where you need to really focus to succeed with this framework. The ultimate tip: Use the pyramid as a guideline to testing layers and NOT a hard and fast rule. You still need to put in the legwork to define and create your own automation framework that works for your DevOps team. 1. Unit tests Unit tests are your entry point, and the main focus of an immature test automation strategy. Automate these tests quickly and abundantly. Unit tests isolate and validate specific features to ensure functionality – think smoke tests, regression tests, and configuration tests. Define the parameters of “units” with your internal testing teams, since most will have varying ideas of what individual units look like. Generally, unit tests should be tests that are already broken down into specific micro-units and are relatively straightforward without any external dependencies (they are mocked or stubbed out). Remember, automated unit tests run in isolation so they can be shifted around an automated test suite or toggled on/off as needed. Combine units to create an automated unit test suite, a collection of certified tests that assures success under normal conditions. A good test suite is stable, manageable, serves a breadth of functionality, and is fast with a clear context for developers. Besides the clear benefit of simplicity and high ROI, automating unit tests identifies functional issues in individual modules allowing devs to quickly fix code. Testing feedback is nearly immediate, and unit tests are often stable from run to run. After identifying unit tests to automate, developers use a test-driven development methodology to approach builds systematically. Dev teams following the test automation pyramid strategy face a major problem with the base layer: developing automated unit tests immediately creates faster test runs, but since this pyramid section is so large, too many automated unit tests that fail can slow down the testing cycle. As your product grows and more features are added, unit tests increase, and the pyramid base gets larger and larger. A solution? Don’t test trivial code, production code, and only automate tests with clear, observable behaviors. Use a test methodology (like TDD) Super simple frameworks like the test automation pyramid benefit from implementing a structure that unifies designing, building, and testing code. Here’s an example of the benefit of using a testing methodology, like test-driven development (TDD). When automating with the TDD methodology, devs automate existing code and only create new code if the automated code fails the test. This strategy quickens the automation process; devs are not creating new scripts for every unit test but working with existing code to automate tests before developing functional build code. Take a deep dive into the basics of functional testing here. Start with the code you have. Run automated unit tests and see what passes and fails to provide results. If the code passes, great! You’ve automated your unit test. If it fails, develop new code and rerun it. Following TDD looks like this: Write. Create a test case module with data. Code. Write just enough production code to fill the test and pass. Refactor. If the code breaks or fails, fix the code as needed locally. TDD is critical to automating unit tests, as it requires devs to think through a specific use case and test before building and shipping code. TDD urges developers to start with small unit tests and write simple scripts to automate (that pass!) before heading back to refactor, a key CI/CD pipeline element. 2. Integration tests The test automation pyramid uses “service tests,” most commonly meaning integration tests. Again, defining what exactly “integration tests” are and the boundaries of integration testing are crucial steps to getting your team automating with intent. Touch on topics like: Are we testing the ENTIRE stack against other applications within the system? Are we going narrow, and testing individual integration points one at a time? Are we only going to automate integration tests we can perform locally? As the second layer of the pyramid, integration tests are a vital yet more complicated process to automate. Once your automated unit testing suite runs smoothly across product and dev teams, you can move toward automating unit integration tests. Don’t assume integration tests can be automated with the same frequency and speed as unit tests. For one, they take longer. There are also a variety of integration components that you need to cover. Are you running integration with specific databases? Filesystems? Internal and external APIs? For help creating an integrated testing strategy, drop us a line. All of that needs to be covered in test cases and test scripts. Still, with so many variables, you need a roster of talented manual testers to oversee integration and validation. Run tests parallel to each other. Once your unit tests are complete and perfected, feedback should take minutes. As that’s being run, a manual tester can start an automated integration test. QE’s can then analyze unit pass/fail and integration pass-fail. If your units are up to date and functional, unit testing will pass while waiting for integration tests to run. If both fail, manual testers can analyze flags and get results back to devs. 3. E2E Process: Separate UI tests from end-to-end If you haven’t noticed, the simplicity of the test automation pyramid makes it difficult to define what each segment is truly referring to. The top of the pyramid, UI tests, are often conflated with end-to-end testing because most UI tests are done by running your app through the E2E process. A pyramid hack is to run UI tests without running E2E tests. Separate out individual UI and write unit tests to test your frontend code with programs like Nunit and Selenium. UI tests are harder to automate: that’s why they sit nestled at the top. For so many teams that decide UI = E2E, automation means notoriously flaky and false-positive results. Automating E2E testing workflows reach a bottleneck when false-positive results need to be manually debugged, again and again. Amidst a talent shortage if you’re team doesn’t have enough skilled quality engineers, automating end-to-end processes is costly, incredibly long, and very fragile. In my test automation pyramid, E2E tasks would be an even smaller tip of the iceberg; use them sparingly, and don’t expect them to provide high ROI, better speed, or be helpful. Instead, center your team on the lower blocks of the pyramid. Extend your definition of unit or integration tests, or improve your test code. By automating individual tests and taking an optimized approach, you can wield the talent to pick tasks with the highest impact potential. Like most agile processes, following the test automation pyramid encourages teams to iterate quickly and efficiently and create smarter systems, but this framework is not without pitfalls. Focus on creating and establishing definitions, boundaries, and best practices that work for your team and create better, smarter processes. Whether you’re looking to implement new automation strategies or improve existing ones, Testlio provides the expertise and tools necessary to achieve efficient and effective quality assurance. Visit our Test Automation Solutions page to learn more about how we can speed up your testing efforts! Discover how Testlio can help you maximize the potential of your test automation.