[Paper Review] Unified Pre-training for Program Understanding and Generation

Wasi Uddin Ahmad, Saikat Chakraborty+2 more|arXiv (Cornell University)|Mar 10, 2021

Software Engineering ResearchComputer Science46 references40 citations

TL;DR

PLBART is a unified sequence-to-sequence pre-trained model on Java, Python, and natural language data that achieves state-of-the-art or competitive results across code summarization, generation, translation, and several discriminative program-understanding tasks.

ABSTRACT

Code summarization and generation empower conversion between programming language (PL) and natural language (NL), while code translation avails the migration of legacy code from one PL to another. This paper introduces PLBART, a sequence-to-sequence model capable of performing a broad spectrum of program and language understanding and generation tasks. PLBART is pre-trained on an extensive collection of Java and Python functions and associated NL text via denoising autoencoding. Experiments on code summarization in the English language, code generation, and code translation in seven programming languages show that PLBART outperforms or rivals state-of-the-art models. Moreover, experiments on discriminative tasks, e.g., program repair, clone detection, and vulnerable code detection, demonstrate PLBART's effectiveness in program understanding. Furthermore, analysis reveals that PLBART learns program syntax, style (e.g., identifier naming convention), logical flow (e.g., if block inside an else block is equivalent to else if block) that are crucial to program semantics and thus excels even with limited annotations.

Motivation & Objective

• Motivate and enable a general-purpose PL-NL understanding and generation model for software engineering (PLUG tasks).
• Leverage unlabeled PL and NL data to learn transferable representations via denoising sequence-to-sequence pre-training.
• Pre-train a multilingual encoder-decoder model on Java, Python, and NL data to support diverse downstream tasks.
• Show that the pre-trained model outperforms or rivals task-specific baselines on generation, translation, and discriminative tasks.

Proposed method

• Adopt a BART-style encoder-decoder transformer with 6 encoder and 6 decoder layers (768 hidden units, 12 attention heads).
• Pre-train using denoising autoencoding on Java, Python functions and StackOverflow NL text with three noising strategies: token masking, token deletion, and token infilling.
• Tokenization via SentencePiece (50k subword units) and language-id tokens to enable multilingual reception.
• Upsample/downsample data across modalities to balance PL and NL data during pre-training.
• Train with a mixed-language sampling scheme and a multinomial sampling distribution to handle imbalanced data, with a total of 100K pre-training steps on multi-GPU setup.
• Fine-tune for sequence generation (summarization, generation, translation) and sequence classification tasks with task-specific inputs and an appended language id, using BLEU, CodeBLEU, EM, and accuracy as evaluation signals.

Experimental results

Research questions

• RQ1 Can a unified PL-NL pre-trained model learn robust representations for both programming languages and natural language?
• RQ2 Does denoising pre-training enable the model to capture program syntax, naming conventions, and data-flow semantics essential for code understanding?
• RQ3 How does a unified model perform on generation, translation, and discriminative programming tasks, especially in languages with limited labeled data?
• RQ4 Does pre-training on large unlabeled PL/NL data yield improvements over encoder-only or decoder-only baselines for PLUG tasks?

Key findings

• PLBART outperforms or rivals state-of-the-art baselines on code summarization, code generation, and code translation across multiple languages.
• PLBART demonstrates strong performance on discriminative tasks such as program repair and vulnerability/clone detection, indicating solid program understanding.
• Ablation studies show PLBART’s ability to learn syntax and data-flow semantics during pre-training, enabling effective fine-tuning with limited labeled data.
• Qualitative analyses indicate that PLBART captures programming constructs, naming conventions, and data-flow patterns that are crucial to program semantics.
• On Ruby (the language with fewest training examples in the set), PLBART shows the largest relative gains, suggesting strong generalization from the unified pre-training approach.