[论文解读] Boosting LLMs for Mutation Generation

Bo Wang, Ming Deng+6 more|arXiv (Cornell University)|Mar 25, 2026

Software Testing and Debugging Techniques被引用 0

一句话总结

SMART 通过整合检索增强生成、代码分块和有监督微调，提升基于 LLM 的变异生成的有效性与效果，小模型达到接近 GPT-4o 的性能水平。

ABSTRACT

LLM-based mutation testing is a promising testing technology, but existing approaches typically rely on a fixed set of mutations as few-shot examples or none at all. This can result in generic low-quality mutations, missed context-specific mutation patterns, substantial numbers of redundant and uncompilable mutants, and limited semantic similarity to real bugs. To overcome these limitations, we introduce SMART (Semantic Mutation with Adaptive Retrieval and Tuning). SMART integrates retrieval-augmented generation (RAG) on a vectorized dataset of real-world bugs, focused code chunking, and supervised fine-tuning using mutations coupled with real-world bugs. We conducted an extensive empirical study of SMART using 1,991 real-world Java bugs from the Defects4J and ConDefects datasets, comparing SMART to the state-of-the-art LLM-based approaches, LLMut and LLMorpheus. The results reveal that SMART substantially improves mutation validity, effectiveness, and efficiency (even enabling small-scale 7B-scale models to match or even surpass large models like GPT-4o). We also demonstrate that SMART significantly improves downstream software engineering applications, including test case prioritization and fault localization. More specifically, SMART improves validity (weighted average generation rate) from 42.89% to 65.6%. It raises the non-duplicate rate from 87.38% to 95.62%, and the compilable rate from 88.85% to 90.21%. In terms of effectiveness, it achieves a real bug detection rate of 92.61% (vs. 57.86% for LLMut) and improves the average Ochiai coefficient from 25.61% to 38.44%. For fault localization, SMART ranks 64 more bugs as Top-1 under MUSE and 57 more under Metallaxis.

研究动机与目标

• Motivate improving mutation generation quality to reflect real-world bugs.
• Develop context-aware mutation generation by leveraging real-world bug data.
• Reduce invalid, duplicate, and uncompilable mutants while increasing semantic relevance.
• Enable smaller models to achieve competitive performance with larger LLMs.
• Demonstrate downstream benefits in test case prioritization and fault localization.

提出的方法

• Build a retrieval-augmented generation (RAG) pipeline over a vectorized bug–fix dataset of 130,000 Java bugs.
• Apply logic-based code chunking to decompose focal methods into semantically coherent chunks.
• Design a task-specific prompt and context integration for LLM-driven mutation generation.
• Fine-tune LLMs with supervised learning on 13,760 mutations coupled with real bugs.
• Evaluate across 1,991 real-world Java defects from Defects4J and ConDefects using multiple models, including 7B and GPT-4o.

Figure 1 . The Overview of Mutation Generation Process of SMART

实验结果

研究问题

• RQ1 RQ1: Does SMART generate more valid mutations than existing approaches?
• RQ2 RQ2: Do SMART mutations more closely resemble real bugs than baselines?
• RQ3 RQ3: How does SMART affect mutation-based test case prioritization performance?
• RQ4 RQ4: How does SMART affect mutation-based fault localization performance?
• RQ5 RQ5: What is the contribution of each SMART component (RAG, chunking, fine-tuning) via ablation?

主要发现

• Validity improvements: weighted average generation rate increases from 42.89% (LLMut) to 65.6%.
• Non-duplicate rate rises from 87.38% (LLMut) and 85.87% (LLMorpheus) to 95.62%.
• Compilable rate improves from 88.85% (LLMut) and 78.43% (LLMorpheus) to 90.21%.
• Effectiveness: real bug detection rate reaches 92.61% (vs. 57.86% for LLMut and 31.99% for LLMorpheus).
• Ochiai coefficient increases to 38.44% (AOC improvement substantial).
• Downstream gains: more Top-1 bug rankings under MUSE (64) and Metallaxis (57); 7B-scale models rival GPT-4o performance.

Figure 2 . The Example Mutation of SMART