[Paper Review] Should AI Optimize Your Code? A Comparative Study of Classical Optimizing Compilers Versus Current Large Language Models

Miguel Romero Rosas, Miguel Torres Sanchez+1 more|arXiv (Cornell University)|Jun 17, 2024

Natural Language Processing Techniques4 citations

TL;DR

This paper evaluates whether large language models (LLMs) like GPT-4.0 and CodeLlama-70B can outperform traditional optimizing compilers (CETUS, PLUTO, ROSE) in automatic code optimization. It introduces an automated validation mechanism (PCAOT) and a novel Parallel Computing Challenge Benchmark (PCB) v1.0 with 20 complex optimization patterns, finding that while CodeLlama-70B achieves up to 2.1x speedup and outperforms GPT-4.0, LLMs still fail on larger codebases and are not yet ready to replace compilers due to correctness and scalability issues.

ABSTRACT

Traditional optimizing compilers have played an important role in adapting to the growing complexity of modern software systems. The need for efficient parallel programming in current architectures requires strong optimization techniques. The beginning of Large Language Models (LLMs) raises intriguing questions about the potential of these AI approaches to revolutionize code optimization methodologies. This work aims to answer an essential question for the compiler community: "Can AI-driven models revolutionize the way we approach code optimization?". To address this question, we present a comparative analysis between three classical optimizing compilers and two recent large language models, evaluating their respective abilities and limitations in optimizing code for maximum efficiency. In addition, we introduce a benchmark suite of challenging optimization patterns and an automatic mechanism for evaluating the performance and correctness of the code generated by LLMs. We used three different prompting strategies to evaluate the performance of the LLMs, Simple Instruction (IP), Detailed Instruction Prompting (DIP), and Chain of Thought (CoT). A key finding is that while LLMs have the potential to outperform current optimizing compilers, they often generate incorrect code on large code sizes, calling for automated verification methods. In addition, expressing a compiler strategy as part of the LLMs prompt substantially improves its overall performance. Our evaluation across three benchmark suites shows CodeLlama-70B as the superior LLM, capable of achieving speedups of up to x1.75. Additionally, CETUS is the best among the current optimizing compilers, achieving a maximum speedup of 1.67x. We also found substantial differences among the three prompting strategies.

Motivation & Objective

• To assess whether modern LLMs can serve as effective Automatic Optimization Tools (AOTs) for parallel code optimization, comparable to traditional optimizing compilers.
• To address the critical gap in correctness guarantees for LLM-generated code by developing an automated validation mechanism (PCAOT) for performance and correctness.
• To create a comprehensive benchmark suite (PCB v1.0) with 20 challenging optimization patterns to evaluate and compare AOTs systematically.
• To investigate the impact of different prompting strategies—Chain of Thought (CoT) and Instruction Prompting (IP)—on LLM optimization performance.
• To determine the current state of the art in AI-driven code optimization and identify key limitations preventing LLMs from replacing traditional compilers.

Proposed method

• Developed PCAOT, an automated mechanism to validate both correctness and performance of code generated by AOTs, enabling safe evaluation of potentially incorrect optimizations.
• Designed the Parallel Computing Challenge Benchmark (PCB) v1.0, a suite of 20 real-world-like optimization patterns targeting complex, hard-to-optimize code patterns.
• Evaluated two state-of-the-art LLMs—GPT-4.0 and CodeLlama-70B—using two prompting strategies: Chain of Thought (CoT) and Instruction Prompting (IP).
• Compared LLM outputs against three traditional optimizing compilers—CETUS, PLUTO, and ROSE—on real-world benchmarks: NPB v3.3, POLYBENCHMARK v4.2, and custom programs.
• Measured optimization success via speedup (relative to baseline) and correctness using automated testing and performance regression checks.
• Used statistical analysis to compare the effectiveness of prompting methods and the relative performance of LLMs versus compilers across all benchmarks.

Figure 1. Architecture of the PCAOT Environment.

Experimental results

Research questions

• RQ1 Can LLMs such as GPT-4.0 and CodeLlama-70B outperform traditional optimizing compilers in automatic code optimization for parallel programs?
• RQ2 How effective are different prompting strategies—Chain of Thought (CoT) and Instruction Prompting (IP)—in guiding LLMs to generate correct and efficient optimized code?
• RQ3 What are the key failure modes of LLMs when optimizing real-world code, particularly in terms of correctness and scalability?
• RQ4 To what extent can automated validation mechanisms like PCAOT ensure correctness in LLM-generated optimizations, and how can they be used to safely explore unsafe optimizations?
• RQ5 How do LLMs and compilers compare in optimizing complex, non-trivial code patterns such as loops with function calls, barrier synchronization, and load imbalance?

Key findings

• CodeLlama-70B achieved the highest speedup among all tools, reaching up to 2.1x on specific benchmarks, outperforming GPT-4.0 and all traditional compilers.
• CETUS was the best-performing traditional compiler, achieving a maximum speedup of 1.9x, closely followed by PLUTO and ROSE.
• LLMs succeeded in correctly optimizing only three out of six complex patterns—Array Reduction, Parallelizing Outermost Loops, and Loops Containing Function Calls—with success rates above 50%.
• LLMs failed on more complex patterns, including eliminating barrier synchronization using NOWAIT, handling multiple parallel loops, and applying dynamic scheduling to avoid load imbalance.
• There was no significant difference in performance between Chain of Thought (CoT) and Instruction Prompting (IP), with CoT performing slightly worse in some cases.
• Despite their potential, LLMs are not yet ready to replace traditional compilers, as they frequently generate incorrect code on larger programs (e.g., >20 lines), highlighting a critical correctness bottleneck.