[Paper Review] Knowledge Transfer from High-Resource to Low-Resource Programming Languages for Code LLMs

Federico Cassano, John Gouwar+8 more|arXiv (Cornell University)|Aug 19, 2023

Software Engineering ResearchComputer Science3 citations

TL;DR

This paper proposes MultiPL-T, a method to enhance Code LLMs for low-resource programming languages by generating high-quality, semi-synthetic training data through test-based code translation from high-resource to low-resource languages. Using a Code LLM to generate and validate tests, then translate code, the approach produces tens of thousands of validated training examples, significantly improving performance on Julia, Lua, OCaml, R, and Racket beyond existing open models.

ABSTRACT

Over the past few years, Large Language Models of Code (Code LLMs) have started to have a significant impact on programming practice. Code LLMs are also emerging as building blocks for research in programming languages and software engineering. However, Code LLMs produce impressive results on programming languages that are well represented in their training data (e.g., Java, Python, or JavaScript), but struggle with low-resource languages that have limited training data available. Low resource languages include OCaml, Racket, and several others. This paper presents an effective approach for boosting the performance of Code LLMs on low-resource languages using semi-synthetic data. Our approach, MultiPL-T, translates training data from high-resource languages into training data for low-resource languages in the following way. 1) We use a Code LLM to synthesize tests for commented code from a high-resource language, filtering out faulty tests and code with low test coverage. 2) We use a Code LLM to translate Python code to a target low-resource language, and use tests to validate the translation. We apply this approach to generate tens of thousands of validated training items for Julia, Lua, OCaml, R, and Racket. Furthermore, we use an open model (StarCoderBase) with open training data (The Stack), which allows us to decontaminate benchmarks, train models without violating licenses, and run experiments that could not otherwise be done. With MultiPL-T generated data, we present fine-tuned versions of StarCoderBase and Code Llama for Julia, Lua, OCaml, R, and Racket. On established benchmarks (MultiPL-E), these models outperform other open Code LLMs. The MultiPL-T approach is easy to apply to new languages, and is significantly more efficient and effective than alternatives such as training longer.

Motivation & Objective

• To address the performance gap in Code LLMs for low-resource programming languages like OCaml, Racket, and Julia, which suffer from insufficient training data.
• To develop a scalable, efficient method for generating high-quality training data for low-resource languages without relying on scarce real-world code.
• To improve fine-tuning efficiency and effectiveness compared to longer training or standard self-instruction approaches.
• To enable open, license-compliant training and benchmarking by using open models and decontaminated data.
• To demonstrate that test-validated translation from high-resource to low-resource languages yields superior performance on established benchmarks.

Proposed method

• Use a Code LLM to generate unit tests for commented code in high-resource languages (e.g., Python), filtering out incorrect or low-coverage tests.
• Train a Code LLM to translate the high-resource code into a low-resource target language, using the generated tests as validation for correctness.
• Apply automated test execution to validate the translated code, ensuring semantic equivalence between source and target.
• Collect and curate tens of thousands of high-quality, validated code-test pairs for low-resource languages like Julia, Lua, OCaml, R, and Racket.
• Fine-tune open-source Code LLMs (StarCoderBase and Code Llama) on the generated semi-synthetic dataset to improve performance.
• Use decontaminated training data from The Stack and open models to ensure reproducibility and compliance with licensing constraints.

Experimental results

Research questions

• RQ1 Can test-validated code translation from high-resource to low-resource languages generate high-quality training data for Code LLMs?
• RQ2 Does fine-tuning on semi-synthetic data from MultiPL-T outperform standard fine-tuning or longer training on low-resource languages?
• RQ3 Can the approach be generalized to multiple low-resource languages with minimal human intervention?
• RQ4 How does the performance of MultiPL-T-fine-tuned models compare to existing open-source Code LLMs on established benchmarks?
• RQ5 Does self-instruction or standard synthetic data generation fail for low-resource languages due to poor code quality?

Key findings

• MultiPL-T-generated datasets significantly improve the performance of StarCoderBase-15B and Code Llama on low-resource languages in the MultiPL-E benchmark.
• Fine-tuned models using MultiPL-T outperform other open-source Code LLMs on all evaluated low-resource languages, including Julia, Lua, OCaml, R, and Racket.
• Training longer on existing low-resource data harms performance, indicating that data quality is more critical than quantity for these languages.
• Self-instruction fails for low-resource languages due to low code quality and incorrect test generation, validating the need for test-based validation.
• The approach is scalable and effective across diverse low-resource languages, demonstrating consistent performance gains.
• The use of open models and decontaminated data enables reproducible, license-compliant experiments that were previously infeasible.