[Paper Review] On the Evaluation of Commit Message Generation Models: An Experimental Study

Wei Tao, Yanlin Wang+6 more|arXiv (Cornell University)|Jul 12, 2021

Software Engineering Research4 citations

TL;DR

This paper conducts a systematic empirical study on commit message generation models, evaluating metrics, datasets, and splitting strategies across multiple programming languages. It identifies BLEU-Norm as the most correlated metric with human judgment, introduces the large-scale MCMD dataset with 3.6M messages in five languages, and demonstrates that model performance varies significantly based on dataset splitting strategies, offering practical guidelines for future evaluation and model selection.

ABSTRACT

Commit messages are natural language descriptions of code changes, which are important for program understanding and maintenance. However, writing commit messages manually is time-consuming and laborious, especially when the code is updated frequently. Various approaches utilizing generation or retrieval techniques have been proposed to automatically generate commit messages. To achieve a better understanding of how the existing approaches perform in solving this problem, this paper conducts a systematic and in-depth analysis of the state-of-the-art models and datasets. We find that: (1) Different variants of the BLEU metric are used in previous works, which affects the evaluation and understanding of existing methods. (2) Most existing datasets are crawled only from Java repositories while repositories in other programming languages are not sufficiently explored. (3) Dataset splitting strategies can influence the performance of existing models by a large margin. Some models show better performance when the datasets are split by commit, while other models perform better when the datasets are split by timestamp or by project. Based on our findings, we conduct a human evaluation and find the BLEU metric that best correlates with the human scores for the task. We also collect a large-scale, information-rich, and multi-language commit message dataset MCMD and evaluate existing models on this dataset. Furthermore, we conduct extensive experiments under different dataset splitting strategies and suggest the suitable models under different scenarios. Based on the experimental results and findings, we provide feasible suggestions for comprehensively evaluating commit message generation models and discuss possible future research directions. We believe this work can help practitioners and researchers better evaluate and select models for automatic commit message generation.

Motivation & Objective

• To address the lack of consistent and comprehensive evaluation practices in commit message generation research.
• To investigate how different evaluation metrics, especially BLEU variants, correlate with human judgment.
• To explore the impact of dataset composition—particularly language diversity and data splitting strategies—on model performance.
• To collect and release a large-scale, multi-language commit message dataset (MCMD) to support broader and more realistic evaluation.
• To provide actionable recommendations for researchers and practitioners on selecting and evaluating commit message generation models.

Proposed method

• Conducted a human evaluation of 100 commit messages with high variance in BLEU scores to assess correlation between automatic metrics and human judgment.
• Proposed and evaluated two improvements to the NNGen retrieval-based model: using n-gram representations (NNGen-Gram4) and applying smoothing functions (NNGen-Smooth).
• Collected and curated the MCMD dataset from 500 repositories across five popular programming languages (Java, Python, JavaScript, C++, Go), containing over 3.6 million commit messages with rich contextual metadata.
• Evaluated state-of-the-art models under three dataset splitting strategies: by commit, by timestamp, and by project, to assess their impact on performance.
• Used correlation coefficients (Pearson and Spearman) to compare BLEU variants (B-Moses, B-Norm, B-CC) against human scores to identify the most reliable metric.
• Reused original model implementations where possible and conducted double-checks on new implementations to ensure reproducibility and validity.

Experimental results

Research questions

• RQ1 Which BLEU variant (B-Moses, B-Norm, B-CC) shows the strongest correlation with human judgment in commit message generation?
• RQ2 How do different dataset splitting strategies (commit-based, timestamp-based, project-based) affect the performance of existing commit message generation models?
• RQ3 To what extent does language diversity in training and evaluation datasets influence model generalization and performance?
• RQ4 Can simple modifications to existing retrieval-based models (e.g., n-gram indexing, smoothing) significantly improve performance?
• RQ5 What are the key limitations and threats in current evaluation practices for commit message generation models?

Key findings

• The BLEU-Norm variant shows the highest correlation with human judgment (Pearson r = 0.62, Spearman ρ = 0.58), making it the most suitable metric for evaluating commit message generation models.
• Model performance varies significantly depending on the dataset splitting strategy: models perform better under commit-based splitting, while performance drops substantially under project-based or timestamp-based splits.
• The MCMD dataset, containing over 3.6 million commit messages across five programming languages, reveals that most existing datasets are heavily skewed toward Java, limiting generalization.
• Improving the retrieval index from 1-gram to 4-gram representations (NNGen-Gram4) increased BLEU-Norm from 23.07 to 23.92, and combining it with smoothing (NNGen-Smooth-Gram4) further raised it to 24.03.
• Existing models show limited generalization under project-based and timestamp-based splits, indicating a need for better data partitioning and data-augmentation strategies.
• Human evaluation revealed significant bias risks, but these were mitigated through clear scoring rules and inter-rater discussions, ensuring low standard deviations among raters.