[Paper Review] Benchmarking and Explaining Large Language Model-based Code Generation: A Causality-Centric Approach

Zhenlan Ji, Pingchuan Ma+2 more|arXiv (Cornell University)|Oct 10, 2023

Software Engineering Research4 citations

TL;DR

This paper proposes a causality-centric framework to evaluate and explain large language model (LLM)-based code generation by modeling causal relationships between natural language prompts and generated code. Using a novel causal graph representation and DiBS-based causal analysis, it identifies how prompt rephrasing affects code quality metrics like BLEU score, revealing actionable insights for improving code generation through prompt engineering.

ABSTRACT

While code generation has been widely used in various software development scenarios, the quality of the generated code is not guaranteed. This has been a particular concern in the era of large language models (LLMs)- based code generation, where LLMs, deemed a complex and powerful black-box model, is instructed by a high-level natural language specification, namely a prompt, to generate code. Nevertheless, effectively evaluating and explaining the code generation capability of LLMs is inherently challenging, given the complexity of LLMs and the lack of transparency. Inspired by the recent progress in causality analysis and its application in software engineering, this paper launches a causality analysis-based approach to systematically analyze the causal relations between the LLM input prompts and the generated code. To handle various technical challenges in this study, we first propose a novel causal graph-based representation of the prompt and the generated code, which is established over the fine-grained, human-understandable concepts in the input prompts. The formed causal graph is then used to identify the causal relations between the prompt and the derived code. We illustrate the insights that our framework can provide by studying over 3 popular LLMs with over 12 prompt adjustment strategies. The results of these studies illustrate the potential of our technique to provide insights into LLM effectiveness, and aid end-users in understanding predictions. Additionally, we demonstrate that our approach provides actionable insights to improve the quality of the LLM-generated code by properly calibrating the prompt.

Motivation & Objective

• To address the lack of systematic evaluation and explanation of LLM-based code generation, especially regarding prompt sensitivity.
• To overcome the opacity of LLMs by identifying causal influences of prompt variations on code quality, rather than relying on black-box metrics.
• To provide developers with actionable, human-understandable insights for improving generated code through prompt calibration.
• To establish a generalizable, language-agnostic framework for causality-based analysis of LLM code generation.

Proposed method

• Proposes a novel causal graph-based representation that maps fine-grained linguistic and code features from prompts and generated code into a structured, analyzable form.
• Quantifies prompts and code outputs using linguistic features (e.g., length, formality) and code features (e.g., complexity, syntax patterns) to enable numerical causal analysis.
• Applies the DiBS algorithm to estimate average treatment effects (ATE) of rephrasing instructions (e.g., 'make it longer', 'be more technical') on code quality metrics.
• Uses ATE estimates to identify which prompt features most significantly influence code quality, enabling causal explanation of model behavior.
• Employs LLM-based rephrasing to generate diverse prompt variants for the same intent, enabling controlled causal analysis across multiple configurations.
• Integrates the framework across three LLMs—GPT-Neo, GPT-3.5-Turbo, and GPT-4—on the APPS dataset to ensure robustness and generalizability.

Experimental results

Research questions

• RQ1 Which prompt rephrasing strategies most significantly improve code generation quality as measured by BLEU and other metrics?
• RQ2 What are the causal relationships between specific linguistic features in prompts and the quality of generated code?
• RQ3 How do different LLMs (GPT-Neo, GPT-3.5-Turbo, GPT-4) respond to the same prompt variations in terms of code quality and stability?
• RQ4 Can causal analysis identify which prompt features lead to high-quality code outputs, and which lead to low-quality or erroneous outputs?
• RQ5 To what extent can prompt calibration, guided by causal insights, improve code generation outcomes across diverse LLMs?

Key findings

• The rephrasing strategy 'make it longer' increased the BLEU score by +0.0531 for GPT-3.5-Turbo and +0.0753 for GPT-4, indicating a statistically significant positive effect on code similarity.
• The 'make it more technical' instruction improved BLEU scores by +0.0297 (GPT-3.5-Turbo) and +0.0362 (GPT-4), showing that formality and technicality in prompts enhance code quality.
• GPT-4 consistently outperformed GPT-3.5-Turbo and GPT-Neo across all rephrasing strategies, with the highest BLEU scores observed under the 'Ours' rephrasing strategy.
• The original prompt often led to lower BLEU scores compared to rephrased versions, suggesting that prompt engineering can significantly improve code generation outcomes.
• Causal analysis revealed that specific linguistic features—such as length and technicality—had measurable and actionable effects on code quality, enabling targeted prompt optimization.
• The framework successfully identified that certain prompt rephrasings led to negative effects (e.g., -0.1367 BLEU for GPT-Neo with 'make it longer'), highlighting the risk of unguided prompt modifications.