[Paper Review] Verified Code Transpilation with LLMs

Sahil Bhatia, Jie Qiu+3 more|arXiv (Cornell University)|Jun 5, 2024

Natural Language Processing Techniques4 citations

TL;DR

This paper introduces LLMLift, an LLM-based framework for verified code transpilation that generates functionally equivalent code in domain-specific languages (DSLs) while automatically producing formal proofs of correctness. By leveraging LLM reasoning to translate source code into a Python-based intermediate representation and then to target DSL syntax, LLMLift achieves 100% correctness across 23 tensor processing benchmarks—outperforming symbolic tools like Tenspiler in both accuracy and speed, without requiring domain-specific heuristics.

ABSTRACT

Domain-specific languages (DSLs) are integral to various software workflows. Such languages offer domain-specific optimizations and abstractions that improve code readability and maintainability. However, leveraging these languages requires developers to rewrite existing code using the specific DSL's API. While large language models (LLMs) have shown some success in automatic code transpilation, none of them provide any functional correctness guarantees on the transpiled code. Another approach for automating this task is verified lifting, which relies on program synthesis to find programs in the target language that are functionally equivalent to the source language program. While several verified lifting tools have been developed for various application domains, they are specialized for specific source-target languages or require significant expertise in domain knowledge to make the search efficient. In this paper, leveraging recent advances in LLMs, we propose an LLM-based approach (LLMLift) to building verified lifting tools. We use the LLM's capabilities to reason about programs to translate a given program into its corresponding equivalent in the target language. Additionally, we use LLMs to generate proofs for functional equivalence. We develop lifting-based compilers for {\em four different} DSLs targeting different application domains. Our approach not only outperforms previous symbolic-based tools in both the number of benchmarks transpiled and transpilation time, but also requires significantly less effort to build.

Motivation & Objective

• To address the challenge of manually rewriting code for domain-specific languages (DSLs), which is error-prone and time-consuming.
• To enable automated, correct-by-construction code transpilation from general-purpose languages to DSLs using large language models (LLMs).
• To eliminate reliance on domain-specific heuristics or complex rule systems used in prior verified lifting tools.
• To demonstrate that LLMs can jointly generate both correct DSL code and formal proofs of functional equivalence.
• To build scalable, reusable lifting compilers for diverse DSLs with minimal engineering effort.

Proposed method

• LLMLift uses a two-phase approach: first, the LLM translates source code into a Python-based intermediate representation (IR) that semantically captures DSL operators.
• Second, the LLM generates the final DSL code from the IR using rewrite rules, guided by prompts that encode DSL semantics.
• The LLM is prompted to simultaneously generate a proof of functional equivalence between the source and transpiled code, using invariants and verification conditions.
• Formal verification is performed using a backend verifier that checks the correctness of the generated proof and code.
• The method relies on prompt engineering to guide the LLM in reasoning about program semantics and generating correct, verifiable outputs.
• The framework is applied to four real-world DSLs: TACO (tensor processing), DSP (digital signal processing), BLAS, and UTDSP, with evaluation on domain-specific benchmarks.

Figure 2 : A high-level overview of our LLMLift framework for building verified lifting-based tools.

Experimental results

Research questions

• RQ1 Can LLMs be effectively used to generate both functionally correct DSL code and formal proofs of equivalence in a single pipeline?
• RQ2 Does an LLM-based approach outperform symbolic verified lifting tools in terms of transpilation accuracy and performance?
• RQ3 Can LLMLift achieve high correctness without requiring custom domain-specific heuristics or optimization rules?
• RQ4 How does joint generation of proof and code compare to sequential or two-phase prompting in terms of reliability?
• RQ5 To what extent can LLMs generalize across diverse DSLs with minimal reconfiguration?

Key findings

• LLMLift successfully transpiled all 23 benchmarks in the tensor processing domain with 100% accuracy, matching or exceeding the performance of Tenspiler, a state-of-the-art symbolic verified lifting tool.
• LLMLift solved all 23 benchmarks without relying on any domain-specific optimizations, whereas Tenspiler with optimizations enabled solved all 23, but failed on 18 when optimizations were removed.
• LLMLift achieved an average transpilation time of 95.89 seconds per benchmark, outperforming Tenspiler’s 115.14 seconds even with all optimizations enabled.
• The single-phase prompting approach (generating proof and invariants together) solved 20 out of 23 benchmarks, indicating that the two-phase method improves reliability by allowing the model to use the generated program state to inform invariant construction.
• LLMLift demonstrated 100% accuracy across all evaluated DSLs, including TACO, DSP, BLAS, mathfu, and UTDSP, across multiple benchmark suites.
• The approach required significantly less engineering effort than prior verified lifting tools, as it does not depend on handcrafted rules or large parallel corpora for training.

Figure 3 : End-to-End Lifting Example