[Paper Review] SynCoBERT: Syntax-Guided Multi-Modal Contrastive Pre-Training for Code Representation

Xin Wang, Yasheng Wang+8 more|arXiv (Cornell University)|Aug 10, 2021

Natural Language Processing TechniquesComputer Science46 references70 citations

TL;DR

SynCoBERT introduces syntax-guided objectives (Identifier Prediction and AST Edge Prediction) and a multi-modal contrastive learning framework to learn code representations from code, comments, and AST, achieving state-of-the-art results on four code intelligence tasks with the same pre-training corpus and model size.

ABSTRACT

Code representation learning, which aims to encode the semantics of source code into distributed vectors, plays an important role in recent deep-learning-based models for code intelligence. Recently, many pre-trained language models for source code (e.g., CuBERT and CodeBERT) have been proposed to model the context of code and serve as a basis for downstream code intelligence tasks such as code search, code clone detection, and program translation. Current approaches typically consider the source code as a plain sequence of tokens, or inject the structure information (e.g., AST and data-flow) into the sequential model pre-training. To further explore the properties of programming languages, this paper proposes SynCoBERT, a syntax-guided multi-modal contrastive pre-training approach for better code representations. Specially, we design two novel pre-training objectives originating from the symbolic and syntactic properties of source code, i.e., Identifier Prediction (IP) and AST Edge Prediction (TEP), which are designed to predict identifiers, and edges between two nodes of AST, respectively. Meanwhile, to exploit the complementary information in semantically equivalent modalities (i.e., code, comment, AST) of the code, we propose a multi-modal contrastive learning strategy to maximize the mutual information among different modalities. Extensive experiments on four downstream tasks related to code intelligence show that SynCoBERT advances the state-of-the-art with the same pre-training corpus and model size.

Motivation & Objective

• Motivate improved code representation by exploiting symbolic and syntactic properties of programming languages.
• Incorporate multiple modalities (code, comments, and AST) to capture complementary information about code semantics.
• Design two novel pre-training objectives (Identifier Prediction and AST Edge Prediction) to encode identifiers and AST edges.
• Introduce a multi-modal contrastive learning mechanism to maximize mutual information across modalities.
• Demonstrate state-of-the-art performance on key code intelligence tasks using the same pre-training corpus and model size.

Proposed method

• Propose SynCoBERT, a 12-layer Transformer encoder pre-trained on code, comments, and AST representations.
• Use Multi-Modal Masked Language Modeling (MMLM) to predict masked tokens across NL, PL, and AST.
• Introduce Identifier Prediction (IP) to classify each code token as identifier or non-identifier.
• Introduce AST Edge Prediction (TEP) to predict the existence of edges between AST node pairs.
• Apply a Multi-Modal Contrastive Learning (MCL) objective to maximize mutual information among code, comments, and AST modalities, using paired (NL-PL-AST) and unpaired data with in-batch and cross-batch negative sampling.
• Aggregate the overall pre-training objective as a combination of MMLM, IP, TEP, MCL, plus L2 regularization.

Experimental results

Research questions

• RQ1 Can syntax-guided objectives (IP and TEP) improve code representations beyond existing MLM-based pre-training?
• RQ2 Does leveraging multi-modal information (code, comments, AST) with contrastive learning lead to more generalizable code representations across tasks?
• RQ3 How does SynCoBERT perform on diverse code intelligence tasks (code search, clone detection, defect detection, program translation) using the same pre-training setup?

Key findings

• SynCoBERT achieves state-of-the-art results on multiple code intelligence tasks using the same pre-training corpus and model size.
• The IP and TEP objectives contribute to performance improvements, demonstrating the value of leveraging symbolic and syntactic properties of code.
• Multi-Modal Contrastive Learning (MCL) provides substantial gains, highlighting the benefit of cross-modality mutual information for code representations.
• SynCoBERT improves natural language code search, code clone detection, code defect detection, and program translation compared to strong baselines such as CodeBERT and GraphCodeBERT.
• Ablation results show that removing IP, TEP, or MCL degrades performance, with MCL having a particularly notable impact on code search.
• The model generalizes to languages not seen during pre-training (e.g., C#/Java translation) due to learned syntactic knowledge.