[Paper Review] LawBench: Benchmarking Legal Knowledge of Large Language Models

Zhiwei Fei, Xiaoyu Shen+7 more|arXiv (Cornell University)|Sep 28, 2023

Artificial Intelligence in LawSocial Sciences19 citations

TL;DR

LawBench evaluates 51 LLMs across 20 Chinese civil-law–oriented tasks to assess legal knowledge memorization, understanding, and applying, revealing that GPT-4 remains the strongest in the legal domain and legal-specific fine-tuning offers limited gains.

ABSTRACT

Large language models (LLMs) have demonstrated strong capabilities in various aspects. However, when applying them to the highly specialized, safe-critical legal domain, it is unclear how much legal knowledge they possess and whether they can reliably perform legal-related tasks. To address this gap, we propose a comprehensive evaluation benchmark LawBench. LawBench has been meticulously crafted to have precise assessment of the LLMs' legal capabilities from three cognitive levels: (1) Legal knowledge memorization: whether LLMs can memorize needed legal concepts, articles and facts; (2) Legal knowledge understanding: whether LLMs can comprehend entities, events and relationships within legal text; (3) Legal knowledge applying: whether LLMs can properly utilize their legal knowledge and make necessary reasoning steps to solve realistic legal tasks. LawBench contains 20 diverse tasks covering 5 task types: single-label classification (SLC), multi-label classification (MLC), regression, extraction and generation. We perform extensive evaluations of 51 LLMs on LawBench, including 20 multilingual LLMs, 22 Chinese-oriented LLMs and 9 legal specific LLMs. The results show that GPT-4 remains the best-performing LLM in the legal domain, surpassing the others by a significant margin. While fine-tuning LLMs on legal specific text brings certain improvements, we are still a long way from obtaining usable and reliable LLMs in legal tasks. All data, model predictions and evaluation code are released in https://github.com/open-compass/LawBench/. We hope this benchmark provides in-depth understanding of the LLMs' domain-specified capabilities and speed up the development of LLMs in the legal domain.

Motivation & Objective

• Assess how much legal knowledge LLMs memorize, including core articles and legal concepts.
• Evaluate understanding of entities, events, and relationships in legal texts.
• Test the ability to apply legal knowledge through realistic reasoning tasks.
• Provide a benchmark to guide development of Chinese-domain legal LLMs.
• Release data, predictions, and evaluation code for reproducibility.

Proposed method

• Propose LawBench with a hierarchical taxonomy (memorization, understanding, applying) based on Bloom’s taxonomy.
• Construct 20 diverse tasks spanning SLC, MLC, regression, extraction, and generation across three cognitive levels.
• Format prompts and implement task-specific answer extraction rules and metrics for fair comparison.
• Evaluate 51 LLMs (multilingual, Chinese-focused, and legal-specialized) on zero-shot and various settings.
• Analyze factors such as SFT, RLHF, model size, and legal-specific fine-tuning to interpret results.
• Provide open-source evaluation code integrated with the OpenCompass platform.

Experimental results

Research questions

• RQ1 How well do LLMs memorize core Chinese legal articles and concepts?
• RQ2 Can LLMs understand complex legal texts, including entities, events, and relationships?
• RQ3 Are LLMs able to apply legal knowledge to realistic tasks requiring reasoning and numerical calculations?
• RQ4 What is the impact of fine-tuning strategies (SFT, RLHF) and model size on legal performance?
• RQ5 How do multilingual and Chinese-oriented LLMs compare to legal-specific LLMs in Chinese civil-law tasks?

Key findings

• GPT-4 remains the best-performing model in the legal domain among those evaluated.
• Legal-specific fine-tuning provides improvements but does not close the gap to top general LLMs in zero-shot settings.
• Fine-tuning benefits vary by task and cognitive level, with general LLMs often outperforming legal-specialized variants in LawBench’s framework.
• LawBench provides extensive data, predictions, and evaluation code for reproducibility and further research.
• The benchmark highlights gaps and guides future development of reliable legal LLMs for the Chinese domain.