[Paper Review] A Prompt Learning Framework for Source Code Summarization

Xu, Tingting, Miao, Yun+9 more|arXiv (Cornell University)|Dec 26, 2023

Software Engineering Research4 citations

TL;DR

This paper proposes PromptCS, a prompt learning framework that trains a continuous prompt agent to generate effective prompts for large language models (LLMs) in source code summarization, eliminating the need for manual prompt engineering. The method achieves state-of-the-art performance comparable to fine-tuning while significantly reducing training costs and improving efficiency, especially on larger LLMs.

ABSTRACT

(Source) code summarization is the task of automatically generating natural language summaries (also called comments) for given code snippets. Recently, with the successful application of large language models (LLMs) in numerous fields, software engineering researchers have also attempted to adapt LLMs to solve code summarization tasks. The main adaptation schemes include instruction prompting, task-oriented (full-parameter) fine-tuning, and parameter-efficient fine-tuning (PEFT). However, instruction prompting involves designing crafted prompts and requires users to have professional domain knowledge, while task-oriented fine-tuning requires high training costs, and effective, tailored PEFT methods for code summarization are still lacking. This paper proposes an effective prompt learning framework for code summarization called PromptCS. It no longer requires users to rack their brains to design effective prompts. Instead, PromptCS trains a prompt agent that can generate continuous prompts to unleash the potential for LLMs in code summarization. Compared to the human-written discrete prompt, the continuous prompts are produced under the guidance of LLMs and are therefore easier to understand by LLMs. PromptCS is non-invasive to LLMs and freezes the parameters of LLMs when training the prompt agent, which can greatly reduce the requirements for training resources. Our comprehensive experimental results show that PromptCS significantly outperforms instruction prompting schemes (including zero-shot learning and few-shot learning) on all four widely used metrics, and is comparable to the task-oriented fine-tuning scheme. In some base LLMs, e.g., StarCoderBase-1B and -3B, PromptCS even outperforms the task-oriented fine-tuning scheme. More importantly, the training efficiency of PromptCS is faster than the task-oriented fine-tuning scheme, with a more pronounced advantage on larger LLMs.

Motivation & Objective

• To address the challenge of manual prompt engineering in code summarization using large language models (LLMs), which requires expert knowledge and is time-consuming.
• To reduce the high computational cost of task-oriented fine-tuning for LLMs in code summarization.
• To develop a non-invasive, parameter-freezing approach that enables effective prompt generation for LLMs without retraining the model.
• To improve the quality and relevance of code summaries by generating prompts that are more compatible with LLMs' understanding.
• To demonstrate that continuous prompt learning can outperform both zero-shot/few-shot instruction prompting and even fine-tuning in some cases.

Proposed method

• A prompt agent is trained to generate continuous prompts instead of discrete, human-crafted prompts, allowing better alignment with LLMs' internal representations.
• The framework keeps LLM parameters frozen during training, making it non-invasive and significantly reducing resource requirements.
• The prompt agent is optimized end-to-end using a loss function based on generated summary quality, guided by LLMs' own outputs.
• Continuous prompts are optimized through gradient-based updates, enabling the model to learn effective prompt patterns automatically.
• The method is evaluated on the CodeSearchNet dataset across multiple programming languages, using standard metrics including BLEU, METEOR, ROUGE-L, and SentenceBERT.
• Human evaluation is conducted to assess the quality and relevance of generated summaries compared to baselines.

Experimental results

Research questions

• RQ1 Can a trainable prompt agent generate continuous prompts that outperform manually designed discrete prompts in code summarization?
• RQ2 Does prompt learning with a frozen LLM achieve performance comparable to task-specific fine-tuning?
• RQ3 How does PromptCS perform across different base LLMs, especially in terms of training efficiency and scalability?
• RQ4 Can continuous prompts improve the quality of code summaries compared to zero-shot or few-shot instruction prompting?
• RQ5 How does the framework perform in human evaluations, and does it generate more natural and accurate summaries?

Key findings

• PromptCS significantly outperforms zero-shot and few-shot instruction prompting across all four metrics: BLEU, METEOR, ROUGE-L, and SentenceBERT.
• On base LLMs such as CodeGen-Multi-2B and StarCoderBase-1B and -3B, PromptCS even surpasses task-oriented fine-tuning in performance.
• The training efficiency of PromptCS is faster than fine-tuning, with a more pronounced advantage on larger LLMs due to parameter freezing.
• Human evaluation confirms that PromptCS generates higher-quality summaries than baseline methods, with improved relevance and naturalness.
• The prompt agent’s ability to generate LLM-compatible continuous prompts is a key factor in the framework’s success.
• The framework demonstrates strong generalization across multiple programming languages on the CodeSearchNet benchmark.