Target and improve three engineering KPIs with better testers Kassidy Kelley , Managing Editor April 28th, 2022 Software dev and QA have a historically complicated relationship. Engineering teams are sprinting to keep up with the astronomical demand for high-quality software and meet target engineering KPIs. But QA delays releases and adds unexpected debugging assignments to already-busy sprint boards. In the words of one engineer from GitLab’s 2021 developer survey, “testing slows down everything.” We’re not taking their quote personally. That dev must have been working with unskilled testers. But when testing is done well, it’s a significant value driver for key software developer KPIs. Better testing doesn’t mean more delayed releases – it means more time for development and less negative attention from product owners because fewer bugs reach production. While traditional crowdsourcing vendors focus heavily on their quantity of testers and unbridled freelancer growth, Testlio takes a more strategic approach. Our platform utilizes a select team of high-quality testers that can move quickly and efficiently. Working with talented testers can improve these three engineering metrics and developers’ lives. 1. Story points completed Working with unskilled third-party QA vendors is time-consuming for dev teams, from the planning stages of test run coordination to bug reproduction. You’ve just sent the test plan for your upcoming release, and now you’re flooded with questions from testers and test managers that don’t understand your product. Or, you’re spending time painstakingly creating a bullet-proof test plan to avoid those last-minute questions in the first place. Reproducing issues once you receive the test report is another time suck for engineering teams. Six percent of dev time is spent reproducing and fixing failed tests – 620 million developer hours each year. If a team of unskilled testers provides unclear bug reports, that number can jump even higher. Testlio’s networked testing team handles test planning, test cases, and run coordination and has managers overseeing independent testing teams. And with highly-vetted testers, the entire process runs more smoothly for engineering teams. Explore the difference between Testlio’s skilled network and unskilled testers. “The quality of [Testlio’s] testers is great. They’re also not changing all the time, so they know the product. I don’t spend time and energy explaining things or spoonfeeding them,” said Remya Ravindran, Senior Quality Engineer at Quizlet. Additionally, these testers write bug reports that devs love with screen recordings, notes from multiple testers across different devices, and concrete steps on how to reproduce issues. Testlio bug reports include what any dev needs to reproduce and fix issues, from device type to video recordings to network conditions or permissions if relevant. Clients say that working with talented testers decreases dev debugging time. “We don’t want our developers spending time figuring out how QA found the issue. Thorough issue reports from Testlio have cut down our debugging time significantly and made us more productive,” said Conrad Kurth, Director of Engineering at Vinovest. Less time spent coordinating test runs and debugging leaves more potential for completed story points and frees up devs for the fun part of their job: writing code. 2. Priority One issues in production A great tester is an engineer’s best friend. They catch issues before they reach production, meaning less frustrated Slack messages from product owners. For Quizlet, their transition from an unskilled to a skilled testing vendor meant finding 25% more P1 issues and avoiding 20 potential crashes per quarter. Their internal incident report Slack channel lit up far less often, and the spotlight on engineering was the right level of shine. Skilled testers understand a product and patterns in software quality, know which buttons to push to uncover issues, and are extremely thorough in pushing them. Testlio’s network has found critical issues like: Technical overlap in Quizlet’s light and dark mode associated with a crash on their web app An incorrect exchange rate on Vinovest’s trading platform that was mispricing wine in a large target market A bug with the camera position functionality on Strava’s mobile app that would have required an emergency dot-release to fix As an engineer, you can quantify this KPI with fewer P1 issues in production, a lower heart rate during the workday, or better sleep the night before a release. But however you want to measure it, you can thank an expert tester when it starts to improve. 3. Fixed issue rates You’ll notice that we only talked about priority one issues in the above paragraph; that’s because expert testers focus on P1 issues to be mindful of engineers’ dev time. Sifting through and triaging issues, especially when they are not high-priority, is another significant time suck for developers. Dev teams get traditional bug hunting results with inexperienced testers – hundreds of low-priority bugs. It’s a quantity over quality approach. With unskilled testers, engineering teams can spend a lot of time triaging issues instead of focusing on core work. George Andraws, a Senior Quality Engineer at SAP Concur suffered through thousands of low-quality bugs when using an unskilled, legacy crowd testing vendor. “There came the point when you’d get an eye roll every time you saw one of their bugs. It was ‘Oh no, here’s another 5 minutes of my time wasted looking into a bug we’re probably not going to fix,'” said Andraws. SAP Concur switched to Testlio’s testing team, and their fixed issue rate tripled. Testlio’s testers are paid hourly instead of per bug, giving them the freedom to focus on the issues that mattered to George’s team, and triaged them in advance, leading to less wasted time on low-priority issues and a higher-quality product for end-users. Tester expertise is vital for engineering teams The difference between a crowd testing approach and a networked testing approach is a focus on quality, not quantity. From the number of freelance testers employed, the number of bugs reported, and the number of priority one issues in production, less is more. The benefactors of great testers are engineering teams who complete more story points, release fewer issues to production, and fix a higher percentage of issues. Tester expertise matters – to engineering teams just as much as end-users.