[Paper Review] CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learning

H Le, Yue Wang+3 more|arXiv (Cornell University)|Jul 5, 2022

Software Engineering ResearchComputer Science87 citations

TL;DR

CodeRL combines a pretrained code LM (CodeT5) with an actor-critic reinforcement learning framework that uses unit-test signals to refine code generation. It achieves state-of-the-art results on APPS and strong zero-shot transfer on MBPP.

ABSTRACT

Program synthesis or code generation aims to generate a program that satisfies a problem specification. Recent approaches using large-scale pretrained language models (LMs) have shown promising results, yet they have some critical limitations. In particular, they often follow a standard supervised fine-tuning procedure to train a code generation model only from the pairs of natural-language problem descriptions and ground-truth programs. Such paradigm largely ignores some important but potentially useful signals in the problem specification such as unit tests, which thus often results in poor performance when solving complex unseen coding tasks. To address the limitations, we propose "CodeRL", a new framework for program synthesis tasks through pretrained LMs and deep reinforcement learning (RL). Specifically, during training, we treat the code-generating LM as an actor network, and introduce a critic network that is trained to predict the functional correctness of generated programs and provide dense feedback signals to the actor. During inference, we introduce a new generation procedure with a critical sampling strategy that allows a model to automatically regenerate programs based on feedback from example unit tests and critic scores. For the model backbones, we extended the encoder-decoder architecture of CodeT5 with enhanced learning objectives, larger model sizes, and better pretraining data. Our method not only achieves new SOTA results on the challenging APPS benchmark, but also shows strong zero-shot transfer capability with new SOTA results on the simpler MBPP benchmark.

Motivation & Objective

• Motivate improving program synthesis beyond standard supervised fine-tuning by leveraging functional signals from unit tests.
• Propose an actor-critic RL framework where the actor is a pretrained code LM and the critic predicts functional correctness.
• Enhance pretraining of CodeT5 with larger data and a next-token prediction objective to better suit generation tasks.
• Introduce a novel inference-time generation procedure that uses unit-test feedback and critic guidance to regenerate or repair programs.

Proposed method

• Treat the code-generating language model as an actor in an RL setting and sample synthetic code sequences.
• Train a critic to predict unit-test outcomes (CompileError, RuntimeError, FailedTest, PassedTest) and use its hidden states to estimate token-level values.
• Define the RL return from unit-test feedback and apply a baseline to stabilize training.
• Incorporate intermediate returns from the critic to provide token-level guidance during generation.
• Implement a critiqued generation procedure at inference time that uses example unit tests and critic scores to refine or repair outputs.

Experimental results

Research questions

• RQ1 How can unit tests be integrated into reinforcement learning to improve functional correctness in program synthesis?
• RQ2 Does an actor-critic framework with a critic predicting unit-test outcomes improve generation compared to standard fine-tuning?
• RQ3 Can extended pretraining of CodeT5 with next-token prediction and larger Python data boost performance on code-generation benchmarks?
• RQ4 What is the impact of critic-guided generation and program repair/refinement during inference on final correctness?
• RQ5 Is the approach transferable across different code-generation models and benchmarks (e.g., APPS, MBPP)?

Key findings

• Achieves state-of-the-art results on APPS with pass@1, pass@5, and pass@1000 improvements (more than 2%, 6%, and 20% respectively).
• Demonstrates strong zero-shot transfer on MBPP, attaining 63.0% pass@80 over a finetuned GPT-3-7B baseline’s 61.4%.
• Extends CodeT5 with larger model sizes and improved pretraining data/objectives, yielding competitive performance with larger LMs.
• RL-based fine-tuning with unit-test signals consistently improves performance across different backbone models.
• The proposed critic sampling procedure enables effective generation, refinement, and repair of programs based on functional correctness signals.