[Paper Review] Large Legal Fictions: Profiling Legal Hallucinations in Large Language Models

Matthew Dahl, Varun Magesh+2 more|arXiv (Cornell University)|Jan 2, 2024

Artificial Intelligence in Law103 references14 citations

TL;DR

The paper analyzes legal hallucinations in LLMs by applying a suite of open-domain, case-law QA tasks, revealing high hallucination rates and limitations in detecting or correcting erroneous legal content.

ABSTRACT

Do large language models (LLMs) know the law? These models are increasingly being used to augment legal practice, education, and research, yet their revolutionary potential is threatened by the presence of hallucinations -- textual output that is not consistent with legal facts. We present the first systematic evidence of these hallucinations, documenting LLMs' varying performance across jurisdictions, courts, time periods, and cases. Our work makes four key contributions. First, we develop a typology of legal hallucinations, providing a conceptual framework for future research in this area. Second, we find that legal hallucinations are alarmingly prevalent, occurring between 58% of the time with ChatGPT 4 and 88% with Llama 2, when these models are asked specific, verifiable questions about random federal court cases. Third, we illustrate that LLMs often fail to correct a user's incorrect legal assumptions in a contra-factual question setup. Fourth, we provide evidence that LLMs cannot always predict, or do not always know, when they are producing legal hallucinations. Taken together, our findings caution against the rapid and unsupervised integration of popular LLMs into legal tasks. Even experienced lawyers must remain wary of legal hallucinations, and the risks are highest for those who stand to benefit from LLMs the most -- pro se litigants or those without access to traditional legal resources.

Motivation & Objective

• Develop a typology of legal hallucinations to frame future research.
• Quantify the prevalence of factual inaccuracies in LLM responses to verifiable legal questions.
• Assess the ability of LLMs to handle contra-factual premises and to gauge their own certainty about answers.
• Investigate how model performance varies with case age, prominence, and court level to identify potential monocultures in legal reasoning.

Proposed method

• Create an original set of 14 legal research tasks increasing in complexity (existence, court, citation, author, disposition, quotation, authority, overruling year, doctrinal agreement, factual background, procedural posture, subsequent history, core legal question, central holding).
• Use reference-based querying with ground-truth metadata from Caselaw Access Project, Supreme Court Database, and other sources to compute population hallucination rates.
• Apply two querying paradigms: reference-based (ground-truth answers) and reference-free (contradiction-based with GPT-4 adjudication).
• Evaluate three LLMs (ChatGPT 3.5, PaLM 2, Llama 2) under zero- and few-shot prompting across four task categories (low, moderate, high complexity).
• Compute hallucination rate as the proportion of incorrect outputs versus ground-truth for reference-based tasks, and a lower bound from the contradiction-based reference-free tasks.

Figure 1: Hallucinations are common across all the LLMs we test when they are asked a direct, verifiable question about a federal court case. (Figure pools all reference-based tasks.)

Experimental results

Research questions

• RQ1 What is the prevalence of factual (open-domain) hallucinations when LLMs answer verifiable legal questions about cases?
• RQ2 How do hallucination rates vary with task complexity, court level, jurisdiction, case prominence, and year across different LLMs?
• RQ3 Are LLMs susceptible to contra-factual bias, and can they reliably assess their own certainty about legal answers?
• RQ4 Do higher-temperature, non-greedy prompts yield more contradictions that indicate non-factual outputs?

Key findings

• Hallucinations are widespread, observed in 69% of queries for ChatGPT 3.5 and 88% for Llama 2 on verifiable federal-case questions.
• Hallucination rates rise with task complexity and are lower for newer, more salient, and more prominent jurisdictions.
• LLMs often provide incorrect answers to contra-factual questions and struggle to gauge their own certainty without post-hoc recalibration.
• Reference-free methods establish a lower bound on hallucination rates, with high-complexity tasks like core legal questions and central holdings showing substantial non-factual outputs.
• Performance is generally better for simpler tasks and higher-quality or more prominent cases, suggesting a legal monoculture effect in LLMs.
• Overall, current LLMs are not reliably trustworthy for essential legal tasks and their use should be cautious, especially for vulnerable users like pro se litigants.

Figure 3: Relationship between task complexity and mean hallucination rate. Higher values indicate a greater likelihood of factually incorrect LLM responses. High complexity tasks include several reference-free tasks, so those reported hallucination rates are lower bounds on the true rates. Contra-f