[Paper Review] Language-Agnostic Representation Learning of Source Code from Structure and Context

Daniel Zügner, Tobias Kirschstein+3 more|arXiv (Cornell University)|Mar 21, 2021

Software Engineering ResearchComputer Science50 references63 citations

TL;DR

The paper introduces the Code Transformer, which jointly learns from source code (Context) and its AST (Structure) using language-agnostic features, achieving state-of-the-art code summarization and enabling multilingual training.

ABSTRACT

Source code (Context) and its parsed abstract syntax tree (AST; Structure) are two complementary representations of the same computer program. Traditionally, designers of machine learning models have relied predominantly either on Structure or Context. We propose a new model, which jointly learns on Context and Structure of source code. In contrast to previous approaches, our model uses only language-agnostic features, i.e., source code and features that can be computed directly from the AST. Besides obtaining state-of-the-art on monolingual code summarization on all five programming languages considered in this work, we propose the first multilingual code summarization model. We show that jointly training on non-parallel data from multiple programming languages improves results on all individual languages, where the strongest gains are on low-resource languages. Remarkably, multilingual training only from Context does not lead to the same improvements, highlighting the benefits of combining Structure and Context for representation learning on code.

Motivation & Objective

• Motivate learning meaningful representations of programs by combining two complementary views: code as text (Context) and its abstract syntax tree (Structure).
• Develop a language-agnostic transformer model that integrates Context and Structure without language-specific preprocessing.
• Demonstrate state-of-the-art performance on monolingual code summarization across five languages.
• Introduce and evaluate a multilingual code summarization model trained on multiple languages using shared vocabularies and language embeddings.

Proposed method

• Adopt a Transformer-based architecture that uses relative distances in attention rather than absolute positions.
• Compute and incorporate multipleRelation distances from the AST (shortest path, ancestor, sibling, and Personalized PageRank) into the attention mechanism.
• Use per-relation key projection matrices to sum contributions from Context and Structure in the attention scores.
• Represent each token by concatenating its token embedding, the embedding of its assigned AST node type, and the token type from the tokenizer.
• Employ a non-trainable sinusoidal encoding for relative distances to enable structure-aware attention on graphs.
• Train with a pointer network to allow the decoder to point to input positions for improved token predictions.

Experimental results

Research questions

• RQ1 Can a single model learn from both Context (code tokens) and Structure (AST) using language-agnostic features?
• RQ2 Does jointly training on multiple programming languages improve code summarization performance, especially for low-resource languages?
• RQ3 What is the impact of including Structure versus Context alone on monolingual and multilingual code summarization?
• RQ4 How do different AST distance measures (shortest path, ancestor, sibling, PageRank) contribute to performance?
• RQ5 Does multilingual training outperform monolingual training across diverse languages when only language-agnostic features are used?

Key findings

• The Code Transformer achieves state-of-the-art code summarization across five languages in monolingual settings.
• Multilingual training substantially improves performance for all languages, with the strongest gains for low-resource languages.
• Context-only multilingual training does not yield the same improvements as combining Structure and Context.
• Ablations show that both Structure and Context contribute to performance, and the pointer network further enhances results.
• Using multiple AST distance measures together yields better results than any single distance alone.