[論文レビュー] Learning Performance-Improving Code Edits

Alexander Shypula, Aman Madaan+8 more|arXiv (Cornell University)|Feb 15, 2023

Software Engineering Research被引用数 24

ひとこと要約

本論文は PIE を作成し、C++ コードの性能改善編集の大規模データセットを作成する。さらに retrieval-based prompting、performance-conditioned generation、self-play fine-tuning が LLM を適応させ、gem5 シミュレータで測定されたコード性能を信頼性高く最適化できることを示し、平均のスピードアップで人間の最高実績を上回る。

ABSTRACT

With the decline of Moore's law, optimizing program performance has become a major focus of software research. However, high-level optimizations such as API and algorithm changes remain elusive due to the difficulty of understanding the semantics of code. Simultaneously, pretrained large language models (LLMs) have demonstrated strong capabilities at solving a wide range of programming tasks. To that end, we introduce a framework for adapting LLMs to high-level program optimization. First, we curate a dataset of performance-improving edits made by human programmers of over 77,000 competitive C++ programming submission pairs, accompanied by extensive unit tests. A major challenge is the significant variability of measuring performance on commodity hardware, which can lead to spurious "improvements." To isolate and reliably evaluate the impact of program optimizations, we design an environment based on the gem5 full system simulator, the de facto simulator used in academia and industry. Next, we propose a broad range of adaptation strategies for code optimization; for prompting, these include retrieval-based few-shot prompting and chain-of-thought, and for finetuning, these include performance-conditioned generation and synthetic data augmentation based on self-play. A combination of these techniques achieves a mean speedup of 6.86 with eight generations, higher than average optimizations from individual programmers (3.66). Using our model's fastest generations, we set a new upper limit on the fastest speedup possible for our dataset at 9.64 compared to using the fastest human submissions available (9.56).

研究の動機と目的

• Provide a dataset and framework to study high-level program optimizations by LLMs.
• Enable reliable, reproducible performance measurement using the gem5 simulator.
• Evaluate prompting and fine-tuning strategies to adapt pre-trained code LLMs for performance optimization.
• Identify effective adaptation techniques that surpass human performance in average speedups.

提案手法

• Curate PIE, a dataset of performance-improving edits (PIE) from CodeNet with execution times annotated via gem5.
• Use the gem5 full-system simulator to obtain deterministic performance measurements.
• Evaluate prompting strategies including instruction prompting, chain-of-thought, dynamic retrieval-based few-shot prompting.
• Explore fine-tuning approaches: high-quality subset, performance-conditioned generation, and synthetic data via self-play.
• Incorporate performance tags to steer optimization toward higher-performance solutions.
• Augment data with synthetic examples generated by LLMs and filtered for novelty and speedup.
• Measure effectiveness using percent of optimized programs, speedup, and correctness across test sets.

実験結果

リサーチクエスチョン

• RQ1 Can large language models be effectively adapted to high-level code optimization tasks using PIE?
• RQ2 Which prompting or fine-tuning strategies best improve performance and correctness when optimizing code?
• RQ3 How do retrieval-based prompting, performance-conditioned generation, and synthetic self-play data compare in driving speedups?
• RQ4 What is the gap between open models and closed models like GPT-3.5 in this setting, and can open models close it with proper adaptation?

主な発見

Scenario	Model	%Opt	Speedup	Correct
Human reference	Best Human	100.00%	4.06	100.00%
Human reference	Same Human	100.00%	3.64	100.00%
All Models, Prompt	gpt-3.5 , FS-CoT	43.78%	1.61	93.15%
Open-Source, Retrieval	codellama 34B	42.16%	2.57	77.92%
Black-Box, Retrieval	gpt4	69.03%	3.56	95.90%
Open-Source, FineTune	codellama 13B-PC	66.60%	5.65	71.08%
Black Box, FineTune	gpt-3.5 , SP	87.68%	6.86	95.11%

• A dataset of 77,967 training pairs from 1,474 problems enables reliable training and evaluation for performance optimization.
• gem5-based evaluation provides deterministic performance measurements, mitigating phantom improvements seen on real hardware.
• Dynamic retrieval-based prompting substantially outperforms baselines, e.g., GPT-3.5 with retrieval achieves high correctness and speedups.
• Fine-tuning with PIE yields strong improvements; performance-conditioned generation markedly boosts optimization performance.
• GPT-3.5 with synthetic self-play data reaches the highest reported average speedup (6.86×) on the test set, exceeding the best human solution (4.06×).
• Open-code models (codellama) with appropriate tuning can approach or match closed models in performance when using the right adaptation strategies.