[Paper Review] Beyond Functional Correctness: Exploring Hallucinations in LLM-Generated Code

Fang Liu, Yang Liu+6 more|arXiv (Cornell University)|Apr 1, 2024

Low-power high-performance VLSI design30 citations

TL;DR

This study analyzes hallucinations in code generated by large language models, develops a taxonomy of 5 hallucination categories, creates HalluCode for evaluation, and shows current models struggle to recognize and mitigate such hallucinations.

ABSTRACT

The rise of Large Language Models (LLMs) has significantly advanced various applications on software engineering tasks, particularly in code generation. Despite the promising performance, LLMs are prone to generate hallucinations, which means LLMs might produce outputs that deviate from users' intent, exhibit internal inconsistencies, or misaligned with the real-world knowledge, making the deployment of LLMs potentially risky in a wide range of applications. Existing work mainly focuses on investigating the hallucination in the domain of Natural Language Generation (NLG), leaving a gap in comprehensively understanding the types, causes, and impacts of hallucinations in the context of code generation. To bridge the gap, we conducted a thematic analysis of the LLM-generated code to summarize and categorize the hallucinations, as well as their causes and impacts. Our study established a comprehensive taxonomy of code hallucinations, encompassing 3 primary categories and 12 specific categories. Furthermore, we systematically analyzed the distribution of hallucinations, exploring variations among different LLMs and benchmarks. Moreover, we perform an in-depth analysis on the causes and impacts of various hallucinations, aiming to provide valuable insights into hallucination mitigation. Finally, to enhance the correctness and reliability of LLM-generated code in a lightweight manner, we explore training-free hallucination mitigation approaches by prompt enhancing techniques. We believe our findings will shed light on future research about code hallucination evaluation and mitigation, ultimately paving the way for building more effective and reliable code LLMs in the future. The replication package is available at https://github.com/Lorien1128/code_hallucination

Motivation & Objective

• Establish a taxonomy of hallucinations that occur in LLM-generated code and relate them to user intent and context.
• Quantify the distribution and co-occurrence of hallucinations across code LLMs and their relation to code correctness.
• Develop HalluCode, a benchmark to evaluate and improve hallucination recognition and mitigation in code generation.
• Provide empirical insights to guide future detection, evaluation, and mitigation strategies for code LLMs.

Proposed method

• Collected 13,968 code snippets from CodeGen, CodeRL, and ChatGPT across HumanEval and DS-1000 datasets; sampled 3,084 for analysis.
• Used a thematic analysis with multiple annotators to derive a taxonomy of hallucinations (5 categories, 19 types).
• Mapped the taxonomy to NLP hallucination categories and analyzed distribution, co-occurrence, and model differences.
• Constructed HalluCode with 5,663 Python tasks derived from Code Alpaca; designed hallucination injection via type-aware scoring and rule-based approaches; conducted evaluation of hallucination recognition and mitigation.
• Evaluated state-of-the-art models (ChatGPT-4, CodeLlama-7B, DeepSeek-Coder-7B) on HalluCode to assess recognition and mitigation capabilities.

Experimental results

Research questions

• RQ1 What are the prevalent types and distributions of hallucinations in code generated by LLMs?
• RQ2 How do hallucination types co-occur and vary across different LLMs?
• RQ3 What is the relationship between hallucinations and functional correctness of generated code?
• RQ4 Can HalluCode effectively benchmark detection and mitigation of code hallucinations across LLMs?

Key findings

• A taxonomy of 5 primary hallucination categories with 19 leaf types; 2,119 hallucinations identified in 3,084 samples.
• Intent Conflicting (32.1%) and Context Deviation (with subtypes Inconsistency 31.8%, Repetition 17.3%, Dead Code 3.2%) are the most common; Knowledge Conflicting accounts for 15.1%.
• Hallucinations can co-occur within a single program; Context Repetition most frequently co-occurs with other types (~1/3 of such cases).
• Only a small fraction of code containing hallucinations passes all tests (≤10%), and some errors are not directly caused by hallucinations, while hallucinations often indicate underlying issues.
• HalluCode contains 5,663 Python tasks for evaluating hallucination recognition and mitigation; open-source models show limited ability to mitigate hallucinations compared to stronger prompts.
• Different models exhibit distinct dominant hallucination types, reflecting model capabilities and training strategies.