Meet Your New Coworker: ChatGPT – Exploring Generative AI in QA

In the past year, generative AI has gone well beyond basics. Experiments with GenAI, such as ChatGPT4, can solve technical problems using mental models and complex systems consisting of specific rules, context, and toolsets. This is not your mother’s chatbot; tech companies that immediately integrated generative AI tools into their workflow report fast and almost universal performance gains.

For the QA industry, generative AI provides significant productivity boosts in generating, testing, and debugging code. At Testlio, our experiments show generative AI can help humans make software testing as much as 15% to 30% faster and 40% more effective. What will this mean for code creation, software testing – and, most critically – the role of humans in the workplace? Let’s dive in.

Accelerating software development with GenAI

Large language models (LLMs) are trained on publicly available datasets, including vast open-source code repositories across different programming languages and frameworks, making them versatile enough to assist in nearly any technical context. In short: they can do most anything.

For QA, AI pair programming is beneficial in explaining new programming languages, paradigms, and tools. This makes completing routine tasks much faster as generative AI coding takes on the role of a coding assistant. Newer LLMs, like the improved ChatGPT 4 and PaLM 2 from Google, are nearing 100% coding accuracy and often showcase expert-level reasoning capabilities. 

However, the quality of generated code still depends on the volume and diversity of training data. A niche context or a brand-new coding language may not produce adequate results.

Regarding implementation, developers prefer to add generative AI to their existing workflow to take advantage of the autocomplete feature, where new lines of code are suggested based on a localized context. AI coding assistants like GitHub Copilot—a fine-tuned AI model essentially ChatGPT for coding—integrate seamlessly with popular IDEs, such as Visual Studio Code and PyCharm. 

Developers can paste lines of code into the default ChatGPT interface from OpenAI to walk them through improving what they’re working on or have it rewrite the code entirely. 

When armed with specific prompts and enough context, a ChatGPT developer can take on tasks at the same speed as developers with many more years of experience without extensive training. While not consistently accurate, it can provide a quicker answer when compared to crowdsourced community responses. ChatGPT can even self-correct its answers when provided with feedback from flagged errors. 

Enhancing efficiency in app testing with GenAI

By leveraging the power of AI in testing processes, businesses can improve the quality and reliability of their software applications and deliver exceptional user experiences.

With the advent of generative AI, software testing can be streamlined for greater efficiency and accuracy through:

• Issue Capture: AI can quickly scan code and analyze application behavior to identify potential bugs.
• Issue Accuracy: AI can analyze vast data to help testers pinpoint potential issues.
• Issue Description: AI can assist in creating more precise and detailed descriptions of the issues identified during testing.
• Issue Escalation: By analyzing patterns in historical data, AI can predict the potential consequences of a bug, helping prioritize issues effectively. 
• Test Case Creation: AI can generate more concise and accurate instructions for creating test cases, reducing the risk of errors.
• Integrated Testing: Aligned issues, identified and prioritized by generative AI, can be automatically escalated to tracking and workflow systems like JIRA. Ticket updates and progress are synced back to Testlio, ensuring seamless collaboration between testers and developers.

The role of human testers in complex testing scenarios

Knowing when to automate test cases can lead to markedly improved results. However, generative AI has limitations when it comes to software testing. Human expertise utilizing specialized knowledge complements the capabilities of generative AI.

There are cases when a human touch is required to fully understand users and execute testing. Automated tools struggle with evaluating UX, UI, and design elements due to their inability to comprehend subjective aspects. Exploratory testing, which uncovers unforeseen issues, is also challenging for generative AI as it relies on patterns drawn from existing data. 

Tests requiring domain or product-specific knowledge are also better suited for human testers. Remember, GenAI can also stand for general – the training data is broad and general and relies on existing data sets. Niche knowledge, app knowledge, or specifics of your user base will be out of range for ChatGPT.

GenAI isn’t going to replace human QA workers – think of it as your newest coworker, and embrace her with welcome arms. She isn’t going anywhere.