[Paper Review] BashExplainer: Retrieval-Augmented Bash Code Comment Generation based on Fine-tuned CodeBERT

Chi Yu, Guang Yang+3 more|arXiv (Cornell University)|Jun 27, 2022

Software Engineering Research4 citations

TL;DR

BashExplainer proposes a retrieval-augmented, two-stage code comment generation method for Bash scripts using fine-tuned CodeBERT. It retrieves semantically and lexically similar code from a repository, fuses the embeddings of the target and retrieved code, and generates comments via a decoder, outperforming all baselines by at least 8.75% on BLEU-4 and 9.29% on METEOR.

ABSTRACT

Developers use shell commands for many tasks, such as file system management, network control, and process management. Bash is one of the most commonly used shells and plays an important role in Linux system development and maintenance. Due to the language flexibility of Bash code, developers who are not familiar with Bash often have difficulty understanding the purpose and functionality of Bash code. In this study, we study Bash code comment generation problem and proposed an automatic method BashExplainer based on two-stage training strategy. In the first stage, we train a Bash encoder by fine-tuning CodeBERT on our constructed Bash code corpus. In the second stage, we first retrieve the most similar code from the code repository for the target code based on semantic and lexical similarity. Then we use the trained Bash encoder to generate two vector representations. Finally, we fuse these two vector representations via the fusion layer and generate the code comment through the decoder. To show the competitiveness of our proposed method, we construct a high-quality corpus by combining the corpus shared in the previous NL2Bash study and the corpus shared in the NLC2CMD competition. This corpus contains 10,592 Bash codes and corresponding comments. Then we selected ten baselines from previous studies on automatic code comment generation, which cover information retrieval methods, deep learning methods, and hybrid methods.

Motivation & Objective

• To address the lack of automatic comment generation for Bash scripts, which are widely used but difficult to understand for non-experts.
• To improve code readability and maintainability by generating high-quality, natural language comments for Bash code.
• To construct a high-quality, large-scale Bash code-comment corpus by merging data from NL2Bash and NLC2CMD.
• To develop a novel retrieval-augmented deep learning framework that combines information retrieval and neural generation for Bash code.

Proposed method

• Fine-tune CodeBERT on a newly constructed Bash code corpus to create a domain-specific Bash encoder.
• Use a hybrid retrieval module based on semantic and lexical similarity to find the most relevant code snippet from a code repository for a given target code.
• Encode both the target code and the retrieved code using the fine-tuned CodeBERT encoder to produce two contextual vector representations.
• Apply normalization and a fusion layer to combine the two vector representations into a single, enriched representation.
• Pass the fused representation through a decoder to generate the natural language comment for the Bash code.
• Train the entire system in a two-stage strategy: first pre-train the encoder on the corpus, then jointly train the retrieval and generation components.

Experimental results

Research questions

• RQ1 Can a retrieval-augmented approach improve the quality of generated comments for Bash scripts compared to pure deep learning methods?
• RQ2 How effective is fine-tuning CodeBERT on a domain-specific Bash corpus for code representation learning?
• RQ3 Does fusing embeddings from the target code and a retrieved similar code lead to better comment generation than using only the target code?
• RQ4 How does BashExplainer compare to state-of-the-art baselines in terms of automatic and human evaluation metrics?
• RQ5 What is the contribution of each component in BashExplainer, as validated through ablation studies?

Key findings

• BashExplainer outperforms all 10 baselines on BLEU-3, BLEU-4, METEOR, and ROUGE-L by at least 8.75%, 9.29%, 4.77%, and 3.86% respectively.
• The ablation study confirms that both the retrieval module and the fusion layer significantly improve performance, validating the design rationale.
• Human evaluation confirms that BashExplainer generates comments that are more accurate and natural than those from baselines.
• The model achieves strong performance even on complex Bash commands involving pipes, loops, and parameter expansions.
• The constructed corpus of 10,592 Bash code-comment pairs is publicly available and serves as a benchmark for future research.
• A browser plugin based on BashExplainer was developed to support real-time code understanding in practice.