[Paper Review] A Neural Architecture for Generating Natural Language Descriptions from Source Code Changes

Pablo Loyola, Edison Marrese-Taylor+1 more|arXiv (Cornell University)|Apr 17, 2017

Software Engineering Research16 references22 citations

TL;DR

This paper proposes an attention-augmented encoder-decoder neural architecture that generates natural language descriptions from source code changes using commit pairs (before/after) and their associated commit messages. The model achieves semantically sound, human-readable summaries in both in-project and cross-project settings, demonstrating feasibility and generalization across diverse open-source projects with BLEU scores up to 34.3 and validation accuracy of 67.15% on full datasets.

ABSTRACT

We propose a model to automatically describe changes introduced in the source code of a program using natural language. Our method receives as input a set of code commits, which contains both the modifications and message introduced by an user. These two modalities are used to train an encoder-decoder architecture. We evaluated our approach on twelve real world open source projects from four different programming languages. Quantitative and qualitative results showed that the proposed approach can generate feasible and semantically sound descriptions not only in standard in-project settings, but also in a cross-project setting.

Motivation & Objective

• To develop an automated method for generating natural language summaries of source code changes, moving beyond static code summarization to capture dynamic program evolution.
• To address the challenge of generating semantically meaningful descriptions from code changes without relying on hand-crafted features, leveraging end-to-end neural learning.
• To evaluate the model’s generalization capability in cross-project settings, where training and testing occur on different software projects.
• To explore whether attention mechanisms can effectively align code modifications with relevant natural language terms in diverse programming contexts.
• To enable automated documentation of code changes and improve developer comprehension of system evolution.

Proposed method

• The model uses an encoder-decoder architecture with an attention mechanism to align source code changes with natural language descriptions.
• The encoder processes the difference between two consecutive code versions (i.e., the commit delta) as input, learning dense representations of code modifications.
• The decoder generates a natural language description token-by-token, conditioned on the context vector from the encoder and attended code tokens.
• The attention mechanism dynamically focuses on relevant code tokens during decoding, improving alignment between code changes and descriptive words.
• The model is trained on triples of (pre-change code, post-change code, natural language commit message) from real-world commits.
• The approach is evaluated on 12 open-source projects across four programming languages, using both in-project and cross-project settings.

Experimental results

Research questions

• RQ1 Can a neural sequence-to-sequence model generate semantically coherent and natural language descriptions from source code changes without hand-engineered features?
• RQ2 How well does the model generalize to cross-project settings where training and test data come from different software projects?
• RQ3 To what extent can the model capture the intent of code changes through attention mechanisms, even when code and natural language modalities differ significantly?
• RQ4 Does the model exhibit rephrasing capabilities and avoid memorization, especially in low-resource or complex change scenarios?
• RQ5 Can the model generate meaningful summaries for multi-file changes when trained on atomic changes?

Key findings

• The model achieved a validation accuracy of 67.15% and a BLEU score of 34.3 on the full dataset for the Guava project, indicating strong performance on complex changes.
• In cross-project evaluation, the model obtained BLEU scores of 14.6 and 18.9 for only-adding and only-removing changes in Python projects, respectively, showing feasibility in zero-shot settings.
• The model demonstrated rephrasing capabilities, as seen in the Theano example where 'fix crash in the new warning message' was summarized as 'Better warning message'.
• Attention heatmaps revealed that the model effectively aligns code tokens with meaningful words in the generated descriptions, such as focusing on 'warning' and 'message'.
• The model showed robustness in generating general, relevant descriptions while avoiding irrelevant terms like specific numbers or names.
• Despite some memorization in cases like youtube-dl, the majority of generated descriptions were semantically well-correlated with reference descriptions.