[Paper Review] Effective Test Generation Using Pre-trained Large Language Models and Mutation Testing

Arghavan Moradi Dakhel, Amin Nikanjam+3 more|arXiv (Cornell University)|Aug 31, 2023

Software Testing and Debugging TechniquesComputer Science3 citations

TL;DR

This paper proposes MuTAP, a framework that enhances test case effectiveness generated by pre-trained large language models (LLMs) by integrating mutation testing to identify and fix weaknesses in test oracles. By using surviving mutants as feedback to iteratively refine prompts, MuTAP improves bug detection, achieving a 94.06% bug detection rate on real buggy code using Llama-2-chat, outperforming both Pynguin and zero-shot/few-shot LLM approaches.

ABSTRACT

One of the critical phases in software development is software testing. Testing helps with identifying potential bugs and reducing maintenance costs. The goal of automated test generation tools is to ease the development of tests by suggesting efficient bug-revealing tests. Recently, researchers have leveraged Large Language Models (LLMs) of code to generate unit tests. While the code coverage of generated tests was usually assessed, the literature has acknowledged that the coverage is weakly correlated with the efficiency of tests in bug detection. To improve over this limitation, in this paper, we introduce MuTAP for improving the effectiveness of test cases generated by LLMs in terms of revealing bugs by leveraging mutation testing. Our goal is achieved by augmenting prompts with surviving mutants, as those mutants highlight the limitations of test cases in detecting bugs. MuTAP is capable of generating effective test cases in the absence of natural language descriptions of the Program Under Test (PUTs). We employ different LLMs within MuTAP and evaluate their performance on different benchmarks. Our results show that our proposed method is able to detect up to 28% more faulty human-written code snippets. Among these, 17% remained undetected by both the current state-of-the-art fully automated test generation tool (i.e., Pynguin) and zero-shot/few-shot learning approaches on LLMs. Furthermore, MuTAP achieves a Mutation Score (MS) of 93.57% on synthetic buggy code, outperforming all other approaches in our evaluation. Our findings suggest that although LLMs can serve as a useful tool to generate test cases, they require specific post-processing steps to enhance the effectiveness of the generated test cases which may suffer from syntactic or functional errors and may be ineffective in detecting certain types of bugs and testing corner cases PUTs.

Motivation & Objective

• To address the limitation of existing LLM-based test generation tools, which focus on code coverage but often fail to detect real bugs.
• To improve the fault-revealing capability of LLM-generated test cases by leveraging mutation testing as a feedback mechanism.
• To enable effective test generation even without natural language descriptions of the Program Under Test (PUT).
• To reduce the number of ineffective test cases with syntactic or functional errors through iterative refinement using mutant feedback.
• To demonstrate that mutation testing can serve as a powerful evaluation and optimization mechanism for LLM-based test generation.

Proposed method

• MuTAP first generates initial test cases using a pre-trained LLM via zero-shot or few-shot prompting based on the PUT and instructions.
• It validates the generated test cases for syntax and return value correctness, filtering out invalid ones.
• Mutation testing is applied to the PUT to generate mutants and identify surviving mutants—those not killed by the initial test suite.
• The surviving mutants are used to augment the original prompt, creating a refined prompt that includes the PUT, initial test case, and mutant-specific failure patterns.
• The LLM is re-prompted with the augmented prompt to generate improved test cases capable of killing the previously surviving mutants.
• The final test cases are validated and ranked based on correctness and effectiveness in killing mutants.

Experimental results

Research questions

• RQ1 Can mutation testing be effectively used to improve the fault-detection capability of LLM-generated test cases?
• RQ2 How does the integration of surviving mutants into the prompt affect the effectiveness of test case generation?
• RQ3 Can MuTAP generate effective test cases without requiring natural language descriptions of the PUT?
• RQ4 How does MuTAP compare to state-of-the-art automated test generation tools (e.g., Pynguin) and zero-shot/few-shot LLM approaches in detecting real bugs?
• RQ5 To what extent can LLMs enhanced with mutation feedback detect corner cases and subtle bugs?

Key findings

• MuTAP achieved a 93.57% Mutation Score (MS) on synthetic buggy code, outperforming all evaluated approaches.
• Using Codex, MuTAP detected 86.72% of real buggy code snippets in a benchmark, significantly improving over baseline methods.
• With Llama-2-chat, MuTAP detected 94.06% of buggy code, surpassing both Pynguin and zero-shot/few-shot LLM techniques.
• MuTAP detected up to 28% more faulty human-written code snippets than existing methods, with 17% remaining undetected by both Pynguin and LLM-based approaches.
• The integration of surviving mutants into the prompt significantly improved test case quality, particularly in detecting subtle and corner-case bugs.
• MuTAP’s iterative refinement process effectively corrected syntactic and functional errors in initial LLM-generated test cases, increasing their validity and effectiveness.