[Paper Review] Neural Code Summarization: How Far Are We?

Ensheng Shi, Yanlin Wang+6 more|arXiv (Cornell University)|Jul 15, 2021

Software Engineering Research57 references4 citations

TL;DR

This paper conducts a systematic evaluation of five state-of-the-art neural code summarization models across three benchmark datasets, revealing critical flaws in BLEU metric usage, the impact of code pre-processing, and dataset characteristics on model performance. It provides actionable guidelines for more reliable evaluation and identifies key challenges and future research directions in automated code summarization.

ABSTRACT

Source code summaries are important for the comprehension and maintenance of programs. However, there are plenty of programs with missing, outdated, or mismatched summaries. Recently, deep learning techniques have been exploited to automatically generate summaries for given code snippets. To achieve a profound understanding of how far we are from solving this problem, in this paper, we conduct a systematic and in-depth analysis of five state-of-the-art neural source code summarization models on three widely used datasets. Our evaluation results suggest that: (1) The BLEU metric, which is widely used by existing work for evaluating the performance of the summarization models, has many variants. Ignoring the differences among the BLEU variants could affect the validity of the claimed results. Furthermore, we discover an important, previously unknown bug about BLEU calculation in a commonly-used software package. (2) Code pre-processing choices can have a large impact on the summarization performance, therefore they should not be ignored. (3) Some important characteristics of datasets (corpus size, data splitting method, and duplication ratio) have a significant impact on model evaluation. Based on the experimental results, we give some actionable guidelines on more systematic ways for evaluating code summarization and choosing the best method in different scenarios. We also suggest possible future research directions. We believe that our results can be of great help for practitioners and researchers in this interesting area.

Motivation & Objective

• To evaluate the reliability and validity of existing neural code summarization models using standardized benchmarks.
• To investigate how variations in BLEU metric calculation and implementation affect performance evaluation.
• To assess the impact of code pre-processing choices on model summarization performance.
• To examine how dataset characteristics such as corpus size, data splitting, and duplication ratio influence model evaluation outcomes.
• To provide actionable guidelines for more systematic evaluation and method selection in code summarization research.

Proposed method

• Systematic evaluation of five state-of-the-art neural code summarization models on three widely used datasets: ASP, Big-Vul, and CodeXGLUE.
• Comprehensive analysis of BLEU metric variants and their impact on performance comparison, including identification of a previously unknown bug in a commonly used software package.
• Ablation studies on code pre-processing techniques to measure their influence on model output quality and evaluation scores.
• Empirical investigation of dataset-level factors, including corpus size, data splitting strategy, and duplication ratio, on model generalization and evaluation stability.
• Statistical analysis and comparison of model performance across different evaluation settings to identify consistent trends and anomalies.
• Proposed guidelines for standardized evaluation protocols based on empirical findings and best practices in model comparison.

Experimental results

Research questions

• RQ1 How do different variants of the BLEU metric affect the reported performance of code summarization models?
• RQ2 To what extent do code pre-processing choices influence the quality and consistency of model-generated summaries?
• RQ3 How do dataset-specific characteristics—such as corpus size, data splitting method, and duplication ratio—affect model evaluation outcomes?
• RQ4 What are the implications of a previously unknown bug in a widely used BLEU implementation for model comparison in code summarization?
• RQ5 What systematic evaluation practices can improve the reliability and reproducibility of code summarization research?

Key findings

• Different variants of the BLEU metric produce significantly varying performance scores, undermining the validity of direct comparisons across studies.
• A previously unknown bug in a commonly used software package for BLEU calculation leads to incorrect metric values, affecting the reliability of reported results.
• Code pre-processing choices, such as tokenization and normalization, have a substantial impact on model performance and must be standardized across experiments.
• Dataset characteristics such as corpus size, data splitting strategy, and duplication ratio significantly influence model evaluation outcomes and should be carefully controlled.
• The current evaluation practices in code summarization research lack consistency and standardization, leading to potentially misleading conclusions.
• The study provides actionable guidelines for more systematic and reliable evaluation of code summarization models in future research.