[Paper Review] Program Synthesis with Large Language Models

Jacob Austin|arXiv (Cornell University)|Aug 16, 2021

Software Engineering Research94 references28 citations

TL;DR

The paper evaluates large Transformer language models (up to 137B params) on Python code synthesis using two benchmarks (MBPP and MathQA-Python), showing size-based performance gains in few-shot and fine-tuning regimes, exploring dialog-based human feedback, and analyzing limitations in semantic grounding and execution prediction.

ABSTRACT

This paper explores the limits of the current generation of large language models for program synthesis in general purpose programming languages. We evaluate a collection of such models (with between 244M and 137B parameters) on two new benchmarks, MBPP and MathQA-Python, in both the few-shot and fine-tuning regimes. Our benchmarks are designed to measure the ability of these models to synthesize short Python programs from natural language descriptions. The Mostly Basic Programming Problems (MBPP) dataset contains 974 programming tasks, designed to be solvable by entry-level programmers. The MathQA-Python dataset, a Python version of the MathQA benchmark, contains 23914 problems that evaluate the ability of the models to synthesize code from more complex text. On both datasets, we find that synthesis performance scales log-linearly with model size. Our largest models, even without finetuning on a code dataset, can synthesize solutions to 59.6 percent of the problems from MBPP using few-shot learning with a well-designed prompt. Fine-tuning on a held-out portion of the dataset improves performance by about 10 percentage points across most model sizes. On the MathQA-Python dataset, the largest fine-tuned model achieves 83.8 percent accuracy. Going further, we study the model's ability to engage in dialog about code, incorporating human feedback to improve its solutions. We find that natural language feedback from a human halves the error rate compared to the model's initial prediction. Additionally, we conduct an error analysis to shed light on where these models fall short and what types of programs are most difficult to generate. Finally, we explore the semantic grounding of these models by fine-tuning them to predict the results of program execution. We find that even our best models are generally unable to predict the output of a program given a specific input.

Motivation & Objective

• Investigate the capability of general-purpose large language models to synthesize short Python programs from natural language descriptions.
• Create two Python code synthesis datasets (MBPP and MathQA-Python) to evaluate synthesis across different linguistic and programmatic challenges.
• Evaluate performance across model scales (244M to 137B parameters) in few-shot and fine-tuning settings.
• Examine the impact of dialog and human feedback on improving synthesized code.
• Assess semantic grounding by attempting to predict program outputs from inputs and analyze limitations.

Proposed method

• Use dense left-to-right decoder-only Transformer language models trained on large, broad web data including code-containing sources.
• Evaluate synthesis with few-shot prompts and with fine-tuning on task-specific datasets (MBPP: 374 fine-tuning examples; MathQA-Python: larger fine-tuning set).
• Generate multiple samples per problem using temperature sampling, execute code to verify functional correctness, and compare against test cases.
• Prompt design experiments include varying the number and choice of in-prompt examples to assess prompt sensitivity and potential prompt-tuning benefits.
• Analyze error typology (runtime, syntax, test-assert failures) and how model size affects error distribution.
• Consider prompt-overlap with pretraining data and perform adversarial-like checks to assess generalization beyond prompt/test constructs.

Experimental results

Research questions

• RQ1 How well do large language models synthesize Python programs from natural language descriptions across MBPP and MathQA-Python?
• RQ2 How does model size impact few-shot and fine-tuned program synthesis performance?
• RQ3 Can dialog and human feedback meaningfully improve synthesis accuracy?
• RQ4 To what extent do models semantically ground code by predicting program outputs for given inputs?
• RQ5 How robust are the generated programs to adversarial or extended test cases beyond the prompt tests?

Key findings

• Synthesis performance scales log-linearly with model size, with the largest models solving up to 59.6% of MBPP problems in few-shot prompts and about 83.8% on MathQA-Python after fine-tuning.
• Fine-tuning generally yields a ~10 percentage point gain across model sizes for MBPP, and larger gains on MathQA-Python with more extensive fine-tuning.
• Dialog-based human feedback can reduce error rates by about half, improving few-shot performance from ~30% to ~65% over four turns of interaction.
• Models tend to generalize to held-out test cases rather than merely regurgitating prompt content, though some failures occur with adversarial test scenarios; pretraining/test overlap with MBPP is relatively minimal.
• Even the best models have limited semantic grounding and cannot reliably predict the execution results of arbitrary inputs for generated programs, indicating a gap between syntactic generation and true understanding.
• BLEU score poorly correlates with synthesis success, and sampling strategy (temperature) significantly affects performance, with greedy decoding being more effective under strict evaluation budgets.