[Paper Review] ComFormer: Code Comment Generation via Transformer and Fusion Method-based Hybrid Code Representation

Guang Yang, Xiang Chen+5 more|arXiv (Cornell University)|Jul 8, 2021

Software Engineering Research4 citations

TL;DR

ComFormer proposes a novel code comment generation model using a Transformer architecture combined with a fusion-based hybrid code representation that integrates lexical and syntactic code features via AST traversal (Sim_SBT) and Byte-BPE for identifier subword tokenization. It achieves state-of-the-art performance on a large Java corpus, outperforming seven SOTA baselines in BLEU, METEOR, and ROUGE-L metrics, with human evaluation confirming high comment quality.

ABSTRACT

Developers often write low-quality code comments due to the lack of programming experience, which can reduce the efficiency of developers program comprehension. Therefore, developers hope that code comment generation tools can be developed to illustrate the functionality and purpose of the code. Recently, researchers mainly model this problem as the neural machine translation problem and tend to use deep learning-based methods. In this study, we propose a novel method ComFormer based on Transformer and fusion method-based hybrid code presentation. Moreover, to alleviate OOV (out-of-vocabulary) problem and speed up model training, we further utilize the Byte-BPE algorithm to split identifiers and Sim_SBT method to perform AST Traversal. We compare ComFormer with seven state-of-the-art baselines from code comment generation and neural machine translation domains. Comparison results show the competitiveness of ComFormer in terms of three performance measures. Moreover, we perform a human study to verify that ComFormer can generate high-quality comments.

Motivation & Objective

• To address the challenge of low-quality or missing code comments due to developer experience and time constraints.
• To improve program comprehension efficiency by automatically generating accurate, functional comments for source code.
• To overcome limitations of existing methods, such as OOV (out-of-vocabulary) issues and poor handling of syntactic structure.
• To develop a robust, end-to-end deep learning model that effectively fuses lexical and syntactic code representations.
• To validate the model’s performance through both automated metrics and human evaluation.

Proposed method

• Employs a Transformer-based encoder-decoder architecture for sequence-to-sequence code comment generation.
• Introduces a hybrid code representation combining token-level (lexical) and AST-based (syntactic) features using Sim_SBT for AST traversal.
• Applies the Byte-BPE algorithm to subword-encode identifiers, reducing the OOV problem during training.
• Explores three fusion strategies—Jointly Encoder, Shared Encoder, and Single Encoder—to combine lexical and syntactic representations.
• Uses a Single Encoder fusion method, which achieves the best performance by jointly learning from both code representations.
• Leverages a large-scale, high-quality Java code-comment dataset (485,812 pairs) from Hu et al. for training and evaluation.

Figure 1: The framework of our proposed method ComFormer

Experimental results

Research questions

• RQ1 Can a Transformer-based model with hybrid code representation outperform existing SOTA methods in code comment generation?
• RQ2 How do different fusion strategies for lexical and syntactic code representations affect model performance?
• RQ3 To what extent does Byte-BPE subword tokenization mitigate the OOV problem in code comment generation?
• RQ4 Does ComFormer generalize well across varying code lengths, especially for longer methods?
• RQ5 How does ComFormer compare to baselines in human evaluation for comment quality and relevance?

Key findings

• ComFormer achieves superior performance across all three standard metrics—BLEU, METEOR, and ROUGE-L—compared to seven SOTA baselines from code comment generation and neural machine translation.
• The Single Encoder fusion strategy yields the best results, outperforming both Jointly and Shared Encoder configurations.
• ComFormer maintains strong performance even on longer code snippets (up to 125 tokens), though performance drops slightly beyond that due to data sparsity.
• Human evaluation confirms that comments generated by ComFormer are rated as more accurate and natural than those from Hybrid-DeepCom, the best-performing baseline.
• The model demonstrates robustness to OOV terms through Byte-BPE, significantly reducing vocabulary-related issues during training and inference.

Figure 2: The AST traversal results of the methods SBT and Sim_SBT