[Paper Review] Source Code Summarization in the Era of Large Language Models

Weisong Sun, Yun Miao+7 more|arXiv (Cornell University)|Jul 9, 2024

Natural Language Processing Techniques4 citations

TL;DR

This paper conducts a comprehensive empirical study on large language model (LLM)-based source code summarization, evaluating prompting techniques, model hyperparameters, and LLM capabilities across programming languages. It finds that GPT-4-based evaluation is most aligned with human judgment, zero-shot prompting often outperforms advanced techniques, and surprisingly, the 7B-parameter CodeLlama-Instruct surpasses GPT-4 in generating semantic and property-focused code summaries.

ABSTRACT

To support software developers in understanding and maintaining programs, various automatic (source) code summarization techniques have been proposed to generate a concise natural language summary (i.e., comment) for a given code snippet. Recently, the emergence of large language models (LLMs) has led to a great boost in the performance of code-related tasks. In this paper, we undertake a systematic and comprehensive study on code summarization in the era of LLMs, which covers multiple aspects involved in the workflow of LLM-based code summarization. Specifically, we begin by examining prevalent automated evaluation methods for assessing the quality of summaries generated by LLMs and find that the results of the GPT-4 evaluation method are most closely aligned with human evaluation. Then, we explore the effectiveness of five prompting techniques (zero-shot, few-shot, chain-of-thought, critique, and expert) in adapting LLMs to code summarization tasks. Contrary to expectations, advanced prompting techniques may not outperform simple zero-shot prompting. Next, we investigate the impact of LLMs' model settings (including top\_p and temperature parameters) on the quality of generated summaries. We find the impact of the two parameters on summary quality varies by the base LLM and programming language, but their impacts are similar. Moreover, we canvass LLMs' abilities to summarize code snippets in distinct types of programming languages. The results reveal that LLMs perform suboptimally when summarizing code written in logic programming languages compared to other language types. Finally, we unexpectedly find that CodeLlama-Instruct with 7B parameters can outperform advanced GPT-4 in generating summaries describing code implementation details and asserting code properties. We hope that our findings can provide a comprehensive understanding of code summarization in the era of LLMs.

Motivation & Objective

• To systematically evaluate the effectiveness of LLM-based code summarization across prompting techniques, model settings, and programming languages.
• To identify the most reliable automated evaluation method for LLM-generated code summaries, especially given their divergence from reference summaries.
• To investigate whether advanced prompting techniques consistently improve code summarization quality compared to simple zero-shot prompting.
• To assess the impact of model hyperparameters (top_p, temperature) on summary quality across different LLMs and programming languages.
• To evaluate the performance of various LLMs in summarizing code across distinct programming language types, including logic programming.

Proposed method

• The study evaluates four LLMs—CodeLlama-Instruct, StarChat-β, GPT-3.5, and GPT-4—on code summarization tasks using a diverse dataset of code snippets across multiple programming languages.
• It compares five prompting techniques: zero-shot, few-shot, chain-of-thought, critique, and expert prompting to assess their impact on summary quality.
• Model hyperparameters (top_p and temperature) are systematically varied to analyze their influence on summary quality, with temperature set to 0.1 or 0 for determinism in evaluation.
• A heuristic rule-based method is used to extract concise code comments from LLM-generated responses, which are then evaluated using multiple automated metrics.
• The study uses GPT-4 as a human-like evaluator to assess summary quality, comparing its results against traditional metrics like BLEU, METEOR, ROUGE-L, and semantic similarity measures.
• Ablation studies are conducted across multiple programming languages to ensure findings are generalizable and not skewed by language-specific artifacts.

Experimental results

Research questions

• RQ1 Which automated evaluation method most closely aligns with human judgment for assessing LLM-generated code summaries?
• RQ2 Do advanced prompting techniques (e.g., chain-of-thought, critique) consistently improve code summarization performance compared to zero-shot prompting?
• RQ3 How do model hyperparameters (top_p and temperature) affect the quality of LLM-generated code summaries across different base LLMs and programming languages?
• RQ4 How do LLMs perform in summarizing code written in different types of programming languages, especially logic programming (LP)?
• RQ5 Can smaller, open-source LLMs like CodeLlama-Instruct outperform larger, proprietary models like GPT-4 in generating specific types of code summaries?

Key findings

• The GPT-4-based evaluation method is most aligned with human judgment, outperforming traditional metrics like BLEU, METEOR, and ROUGE-L in assessing summary quality.
• Advanced prompting techniques such as chain-of-thought and critique do not consistently outperform simple zero-shot prompting, which often yields superior results.
• The impact of top_p and temperature on summary quality is similar across LLMs and programming languages, though the magnitude of impact varies by base model and language.
• LLMs perform suboptimally when summarizing logic programming (LP) code, showing significantly lower quality compared to procedural and object-oriented languages.
• CodeLlama-Instruct with 7B parameters outperforms GPT-4 in generating 'Why' (4.29 vs. 4.28) and 'Property' (4.19 vs. 4.06) summaries, indicating strong capability in capturing semantic and formal properties.
• The study demonstrates that open-source, smaller LLMs can achieve performance comparable to or exceeding that of large commercial models in specific summarization tasks, especially for semantic and property-focused summaries.