[Paper Review] CoTran: An LLM-based Code Translator using Reinforcement Learning with Feedback from Compiler and Symbolic Execution

Prithwish Jana, Piyush Jha+4 more|arXiv (Cornell University)|Jun 11, 2023

Software Engineering Research4 citations

TL;DR

CoTran is a novel LLM-based code translator that uses reinforcement learning with feedback from compilers and symbolic execution to improve translation correctness. By integrating compiler feedback (for compilation accuracy) and symbolic execution feedback (for functional equivalence), CoTran achieves 48.68% functional equivalence accuracy and 76.98% compilation accuracy on Python-to-Java translation, outperforming prior methods including PLBART-base and CodeT5-based models.

ABSTRACT

In this paper, we present an LLM-based code translation method and an associated tool called CoTran, that translates whole-programs from one high-level programming language to another. Existing LLM-based code translation methods lack training to ensure that the translated code reliably compiles or bears substantial functional equivalence to the input code. In our work, we fine-tune an LLM using reinforcement learning, incorporating compiler feedback, and symbolic execution (symexec)-based testing feedback to assess functional equivalence between the input and output programs. The idea is to guide an LLM during fine-tuning, via compiler and symexec-based testing feedback, by letting it know how far it is from producing perfect translations. We conduct extensive experiments comparing CoTran with 14 other code translation tools, including human-written transpilers, LLM-based translation tools, and ChatGPT. Using a benchmark of over um{57000} code pairs in Java and Python, we demonstrate that CoTran outperforms the other tools on relevant metrics such as compilation accuracy (CompAcc) and functional equivalence accuracy (FEqAcc). For example, in Python-to-Java translation, CoTran achieves 48.68% FEqAcc and 76.98% CompAcc, whereas the nearest competing tool (PLBART-base) gets 38.26% and 75.77% respectively. Additionally, CoTran, built on top of CodeT5, improves FEqAcc by +14.89% and CompAcc by +8.14% for Python-to-Java (resp., +12.94% and +4.30% for Java-to-Python).

Motivation & Objective

• To address the lack of reliable compilation and functional equivalence in existing LLM-based code translation tools.
• To improve whole-program code translation by incorporating corrective feedback during LLM fine-tuning.
• To enhance translation correctness through reinforcement learning with compiler and symbolic execution feedback.
• To enable automated functional equivalence checking via back-to-back LLMs trained on bidirectional translation.
• To develop a parameter-efficient, resource-conscious training framework using LoRA for joint fine-tuning of forward and backward models.

Proposed method

• CoTran uses a two-stage training pipeline combining supervised fine-tuning (SFT) and reinforcement learning (RL) via Proximal Policy Optimization (PPO).
• The method employs compiler feedback (CF) to assess compilation success and symbolic execution feedback (SF) to verify functional equivalence using unit tests generated by Symflower.
• A back-to-back (b2b) architecture uses a Java-to-Python (J2P) and Python-to-Java (P2J) LLM pair to enable end-to-end validation of functional equivalence.
• Feedback rewards are computed as a weighted sum of SF (functional equivalence) and CF (compilation success), with KL-divergence regularization to prevent distributional drift.
• Parameter-efficient fine-tuning is achieved via Low-Rank Adaptation (LoRA), freezing most model parameters and only training low-rank projection matrices.
• An interleaved training loop alternates between SFT and RL optimization, selecting the best-performing model based on validation accuracy.

(a) The baseline CoTran (using Supervised Fine-Tuning i.e. SFT)

Experimental results

Research questions

• RQ1 Can reinforcement learning with compiler and symbolic execution feedback significantly improve the compilation and functional correctness of LLM-based code translation?
• RQ2 How does the integration of bidirectional feedback (from both compiler and symbolic execution) affect translation quality compared to supervised fine-tuning alone?
• RQ3 To what extent does using back-to-back LLMs improve functional equivalence checking in whole-program code translation?
• RQ4 Can parameter-efficient fine-tuning with LoRA maintain high performance while reducing computational cost in joint translation model training?
• RQ5 How does CoTran compare to state-of-the-art tools like PLBART and CodeT5 in terms of compilation accuracy and functional equivalence accuracy?

Key findings

• CoTran achieves 48.68% functional equivalence accuracy (FEqAcc) and 76.98% compilation accuracy (CompAcc) on Python-to-Java translation, significantly outperforming the nearest competitor PLBART-base (38.26% FEqAcc, 75.77% CompAcc).
• When built upon CodeT5, CoTran improves FEqAcc by 11.23% and CompAcc by 4.07% for Java-to-Python translation, and by 14.89% and 8.14% respectively for Python-to-Java translation.
• The use of both compiler feedback and symbolic execution feedback leads to a substantial improvement in translation correctness compared to supervised fine-tuning alone.
• The interleaved SFT-RL training loop with model selection based on validation performance stabilizes training and improves final model quality.
• The parameter-efficient LoRA-based fine-tuning approach enables joint training of forward and backward models with reduced memory and compute requirements.
• The back-to-back LLM setup enables automated, end-to-end functional equivalence checking by translating back and forth between languages.

(b) CoTran + CF (using RL-based fine-tuning by PPO Algorithm)