[論文レビュー] Teaching Large Language Models to Self-Debug

Xinyun Chen, Maxwell Lin+2 more|arXiv (Cornell University)|Apr 11, 2023

Software Engineering Research被引用数 72

ひとこと要約

SELF-DEBUGGING は few-shot prompt を通じて自分自身が生成したコードをデバッグするよう LLM を学習させ、コード実行と自然言語による説明を用いて誤りを反復的に修正することで、Spider (text-to-SQL)、TransCoder (code translation)、MBPP (text-to-Python) で最先端の結果を、より良いサンプル効率で達成します。

ABSTRACT

Large language models (LLMs) have achieved impressive performance on code generation. However, for complex programming tasks, generating the correct solution in one go becomes challenging, thus some prior works have designed program repair approaches to improve code generation performance. In this work, we propose Self-Debugging, which teaches a large language model to debug its predicted program via few-shot demonstrations. In particular, we demonstrate that Self-Debugging can teach the large language model to perform rubber duck debugging; i.e., without any human feedback on the code correctness or error messages, the model is able to identify its mistakes by investigating the execution results and explaining the generated code in natural language. Self-Debugging achieves the state-of-the-art performance on several code generation benchmarks, including the Spider dataset for text-to-SQL generation, TransCoder for C++-to-Python translation, and MBPP for text-to-Python generation. On the Spider benchmark where there are no unit tests to verify the correctness of predictions, Self-Debugging with code explanation consistently improves the baseline by 2-3%, and improves the prediction accuracy on problems of the hardest level by 9%. On TransCoder and MBPP where unit tests are available, Self-Debugging improves the baseline accuracy by up to 12%. Meanwhile, by leveraging feedback messages and reusing failed predictions, Self-Debugging notably improves sample efficiency, and can match or outperform baseline models that generate more than 10x candidate programs.

研究の動機と目的

• LLM が追加の訓練なしで自身の出力をデバッグできるようにすることで、コード生成の改善を促す。
• 生成、実行、説明、フィードバックを用いた自己デバッグフレームワークを提案し、コードを反復的に修正する。
• 複数のコード生成タスクと LLM にわたってアプローチを評価し、性能とサンプル効率の向上を確立する。

提案手法

• Three-step SELF-DEBUGGING loop: Generation of candidate programs, Explanation of the code, and Feedback on correctness.
• Few-shot prompting to provide demonstrations and guide the model through debugging iterations.
• Execution-based selection of final predictions among multiple samples when execution results are available.
• Utilization of unit tests to filter/verify candidate programs when provided in the problem description.
• Delivery of feedback formats including Simple Feedback, Unit Test (UT) Feedback, Code Explanation (Expl), and Execution Trace (Trace); style analogous to rubber-duck debugging.
• Iterative refinement until feedback indicates correctness or a maximum number of turns is reached.

実験結果

リサーチクエスチョン

• RQ1 Can SELF-DEBUGGING improve code-generation accuracy on text-to-SQL, code translation, and text-to-Python tasks across different LLMs?
• RQ2 How do different feedback formats (simple, explanations, execution traces, unit-test integration) affect performance?
• RQ3 What is the impact of unit tests on SELF-DEBUGGING effectiveness and sample efficiency?
• RQ4 How many debugging turns are typically needed, and does execution feedback drive improvements?

主な発見

• Achieves state-of-the-art performance on multiple benchmarks: Spider (text-to-SQL), TransCoder (C++-to-Python translation), and MBPP (text-to-Python).
• On Spider (no unit tests), code explanation with SELF-DEBUGGING improves accuracy by about 2–3% and up to 9% on the hardest queries.
• On TransCoder and MBPP (with unit tests), SELF-DEBUGGING improves baseline accuracy by up to about 12%.
• Code explanation alone provides consistent gains (roughly 2–3%) across tasks; adding unit-test feedback often yields larger gains.
• SELF-DEBUGGING enhances sample efficiency, matching or outperforming baselines that generate more than 10× candidate programs.
• The benefits vary by model; execution feedback (Trace) generally helps, and the approach is effective across Codex, GPT-3.5, GPT-4, and StarCoder, though gains differ by task and model.
• Execution-based selection of final predictions contributes to performance, particularly when multiple samples are available.