[Paper Review] An Extractive-and-Abstractive Framework for Source Code Summarization

Weisong Sun, Chunrong Fang+7 more|arXiv (Cornell University)|Jun 15, 2022

Software Engineering Research4 citations

TL;DR

This paper proposes EACS, a novel extractive-and-abstractive framework for source code summarization that combines deep learning-based statement extraction with abstractive generation. By jointly processing the full code snippet and extracted important statements, EACS generates more natural, informative, and factually accurate summaries than state-of-the-art methods across six programming languages and three datasets, outperforming existing techniques on BLEU, METEOR, and ROUGE-L metrics while demonstrating superior human-rated quality.

ABSTRACT

(Source) Code summarization aims to automatically generate summaries/comments for a given code snippet in the form of natural language. Such summaries play a key role in helping developers understand and maintain source code. Existing code summarization techniques can be categorized into extractive methods and abstractive methods. The extractive methods extract a subset of important statements and keywords from the code snippet using retrieval techniques, and generate a summary that preserves factual details in important statements and keywords. However, such a subset may miss identifier or entity naming, and consequently, the naturalness of generated summary is usually poor. The abstractive methods can generate human-written-like summaries leveraging encoder-decoder models from the neural machine translation domain. The generated summaries however often miss important factual details. To generate human-written-like summaries with preserved factual details, we propose a novel extractive-and-abstractive framework. The extractive module in the framework performs a task of extractive code summarization, which takes in the code snippet and predicts important statements containing key factual details. The abstractive module in the framework performs a task of abstractive code summarization, which takes in the entire code snippet and important statements in parallel and generates a succinct and human-written-like natural language summary. We evaluate the effectiveness of our technique, called EACS, by conducting extensive experiments on three datasets involving six programming languages. Experimental results show that EACS significantly outperforms state-of-the-art techniques in terms of all three widely used metrics, including BLEU, METEOR, and ROUGH-L.

Motivation & Objective

• To address the limitations of existing extractive and abstractive code summarization methods, which either lack naturalness or miss factual details.
• To improve code summarization by combining the factual precision of extractive methods with the fluency of abstractive methods.
• To develop a scalable, modular framework that integrates both extractive and abstractive components for enhanced summary quality.
• To evaluate the framework on diverse datasets across multiple programming languages to ensure generalization and robustness.

Proposed method

• The extractive module uses a deep learning-based classifier to identify and extract statements containing key factual details from the input code snippet.
• The abstractive module takes both the full code snippet and the extracted important statements as parallel inputs to generate a natural language summary.
• The framework employs a dual-encoder architecture where the code and important statements are encoded separately before fusion for summary generation.
• The abstractive module leverages sequence-to-sequence models with attention mechanisms to generate fluent, human-like summaries.
• The framework is designed to be modular and scalable, allowing integration with various pre-trained models or future large language models (LLMs).
• The model is trained end-to-end using sequence generation loss, optimizing for both factual completeness and natural language fluency.

Figure 1. Example of Code Snippet and Summary

Experimental results

Research questions

• RQ1 Can a hybrid extractive-and-abstractive framework improve the factual accuracy and naturalness of code summaries compared to purely extractive or abstractive methods?
• RQ2 How does the joint processing of full code and extracted statements affect summary quality across multiple programming languages?
• RQ3 To what extent does EACS outperform state-of-the-art code summarization techniques on standard evaluation metrics like BLEU, METEOR, and ROUGE-L?
• RQ4 How do human evaluators rate the informativeness, relevance, and naturalness of summaries generated by EACS compared to existing methods?

Key findings

• EACS significantly outperforms state-of-the-art methods on all three standard metrics—BLEU, METEOR, and ROUGE-L—across six programming languages and three datasets.
• Human evaluation confirms that EACS-generated summaries are more natural, informative, and relevant to the input code snippets than those from existing techniques.
• The extractive module effectively identifies key statements containing factual details, improving the abstractive module's ability to generate accurate summaries.
• The abstractive module, when fed both the full code and extracted statements, generates summaries that are more fluent and contextually coherent than those from models using only the code snippet.
• The framework demonstrates strong generalization across diverse programming languages and code structures, indicating broad applicability.
• The results suggest that integrating extractive and abstractive components in a unified framework is more effective than using either approach in isolation.

Figure 2. Framework of DL-based Code Summarization