[Paper Review] A Convolutional Neural Network for Language-Agnostic Source Code Summarization

Jessica Moore, Ben Gelman+1 more|arXiv (Cornell University)|Mar 29, 2019

Software Engineering Research35 references4 citations

TL;DR

This paper proposes a language-agnostic, character-level convolutional neural network for source code summarization that uses an open-vocabulary decoder to generate descriptive comments without requiring language-specific parsing or fixed vocabularies. The model achieves state-of-the-art performance on a single-language (Java) dataset and delivers the first results on a multi-language dataset, demonstrating effective cross-language generalization and robustness to code syntax variations.

ABSTRACT

Descriptive comments play a crucial role in the software engineering process. They decrease development time, enable better bug detection, and facilitate the reuse of previously written code. However, comments are commonly the last of a software developer's priorities and are thus either insufficient or missing entirely. Automatic source code summarization may therefore have the ability to significantly improve the software development process. We introduce a novel encoder-decoder model that summarizes source code, effectively writing a comment to describe the code's functionality. We make two primary innovations beyond current source code summarization models. First, our encoder is fully language-agnostic and requires no complex input preprocessing. Second, our decoder has an open vocabulary, enabling it to predict any word, even ones not seen in training. We demonstrate results comparable to state-of-the-art methods on a single-language data set and provide the first results on a data set consisting of multiple programming languages.

Motivation & Objective

• To address the critical problem of sparse or missing documentation in software development, which increases maintenance costs and hinders code comprehension.
• To overcome the limitations of existing source code summarization models that require language-specific parsers and suffer from closed-vocabulary constraints.
• To develop a unified, language-agnostic model capable of summarizing source code regardless of programming language or syntax.
• To enable the generation of descriptive, natural language comments even for rare or domain-specific terms not seen during training.
• To provide the first empirical results on multi-language code summarization, establishing new baselines for C++ and Python.

Proposed method

• The model employs a deep convolutional encoder that processes source code as a raw sequence of characters, eliminating the need for lexing or parsing.
• Character embeddings are learned end-to-end, allowing the encoder to capture syntactic and semantic patterns across diverse programming languages.
• The decoder uses a novel open-vocabulary mechanism that combines words, subwords, and individual characters to generate any word, including out-of-vocabulary terms.
• The model is trained using a sequence-to-sequence framework with attention, enabling alignment between code features and summary tokens.
• A custom vocabulary construction method is introduced to handle the long-tailed distribution of terms in code comments, improving generalization to rare or project-specific terms.
• The architecture avoids language-specific preprocessing, enabling direct input of raw code, including syntactically incorrect or malformed code.

Experimental results

Research questions

• RQ1 Can a single neural model effectively summarize source code across multiple programming languages without language-specific preprocessing?
• RQ2 How does an open-vocabulary decoder improve the generation of rare or domain-specific terms in code comments compared to closed-vocabulary models?
• RQ3 To what extent can a character-level encoder learn meaningful representations of code functionality independent of syntax or language?
• RQ4 Does the model achieve performance comparable to state-of-the-art language-specific models on a single-language benchmark?
• RQ5 Can the model generalize to previously unseen programming languages, as evidenced by performance on a multi-language dataset?

Key findings

• The model achieves state-of-the-art performance on the Java-based CodeXGLUE benchmark, outperforming prior language-specific models in both BLEU and ROUGE scores.
• The model produces the first reported results on a multi-language code summarization dataset, demonstrating effective zero-shot generalization across C++, Java, and Python.
• The open-vocabulary decoder successfully generates previously unseen terms such as 'playerID' and 'isValid' without relying on unknown token replacement.
• The model maintains high performance even on code with syntactic errors or non-standard formatting, confirming its robustness to input noise.
• The character-level encoder generalizes well across languages, showing that syntactic differences do not hinder semantic understanding.
• The model's ability to generate meaningful comments for rare or project-specific terms confirms the effectiveness of its hybrid word/subword/character prediction mechanism.