[Paper Review] LAiW: A Chinese Legal Large Language Models Benchmark

Y. S. Dai, Duanyu Feng+7 more|arXiv (Cornell University)|Oct 9, 2023

Artificial Intelligence in Law4 citations

TL;DR

This paper introduces LAiW, the first comprehensive benchmark for evaluating Chinese Legal Large Language Models (LLMs) based on three tiers of legal capabilities: basic legal NLP, basic legal application, and complex legal application. The benchmark evaluates 8 models, including legal-specific and general LLMs, revealing that while some legal LLMs outperform their backbones, only ChatGPT achieves state-of-the-art performance across tasks, highlighting gaps in current legal LLMs' reasoning and accuracy.

ABSTRACT

General and legal domain LLMs have demonstrated strong performance in various tasks of LegalAI. However, the current evaluations of these LLMs in LegalAI are defined by the experts of computer science, lacking consistency with the logic of legal practice, making it difficult to judge their practical capabilities. To address this challenge, we are the first to build the Chinese legal LLMs benchmark LAiW, based on the logic of legal practice. To align with the thinking process of legal experts and legal practice (syllogism), we divide the legal capabilities of LLMs from easy to difficult into three levels: basic information retrieval, legal foundation inference, and complex legal application. Each level contains multiple tasks to ensure a comprehensive evaluation. Through automated evaluation of current general and legal domain LLMs on our benchmark, we indicate that these LLMs may not align with the logic of legal practice. LLMs seem to be able to directly acquire complex legal application capabilities but perform poorly in some basic tasks, which may pose obstacles to their practical application and acceptance by legal experts. To further confirm the complex legal application capabilities of current LLMs in legal application scenarios, we also incorporate human evaluation with legal experts. The results indicate that while LLMs may demonstrate strong performance, they still require reinforcement of legal logic.

Motivation & Objective

• To address the lack of a standardized, comprehensive evaluation framework for Chinese legal LLMs.
• To define and structure legal capabilities into three progressive levels: basic legal NLP, basic legal application, and complex legal application.
• To evaluate existing Chinese legal and general LLMs on the first phase of the benchmark, focusing on basic legal NLP tasks.
• To provide open-source evaluation data and instruction tuning datasets for future model development and fine-tuning.
• To ensure ethical integrity by masking sensitive data and involving legal experts in dataset curation.

Proposed method

• The benchmark is structured into three levels of legal capability: basic legal NLP (e.g., text classification, entity recognition), basic legal application (e.g., legal article recommendation), and complex legal application (e.g., judicial summarization, case reasoning).
• Legal and AI experts collaboratively reorganized and classified existing LegalAI tasks into specific, evaluable tasks within each capability level.
• The first evaluation phase focuses on basic legal NLP, including tasks like legal article recommendation, element recognition, judicial summarization, and case recognition.
• Evaluation metrics include accuracy, F1-score, Matthew’s correlation coefficient (MCC), and ROUGE scores for summarization tasks.
• Datasets were constructed with ethical safeguards: personal names, locations, and IDs were masked, and legal experts reviewed data for compliance.
• Instruction tuning datasets were reserved to support future fine-tuning of LLMs on legal tasks.

Experimental results

Research questions

• RQ1 How do Chinese legal LLMs compare to their general LLM backbones in basic legal NLP tasks?
• RQ2 To what extent do legal LLMs improve upon general LLMs in legal-specific capabilities like legal article recommendation and case recognition?
• RQ3 Can general-purpose LLMs like ChatGPT outperform specialized legal LLMs in basic legal NLP tasks despite not being fine-tuned on legal data?
• RQ4 What are the key performance gaps in current legal LLMs when evaluated on structured legal capabilities?
• RQ5 How effective are instruction tuning and retrieval-augmented generation in enhancing legal reasoning and accuracy?

Key findings

• ChatGPT achieved the highest performance across all basic legal NLP tasks, with an accuracy of 0.988 on legal article recommendation and 0.989 on case recognition.
• Baichuan2 and Ziya-LLaMA showed strong performance on case recognition (accuracy: 0.970 and 0.947, respectively), outperforming several legal LLMs.
• HanFei, despite being pre-trained on legal data, underperformed with only 0.169 accuracy on legal article recommendation, suggesting that domain pre-training alone is insufficient.
• LawGPT performed worse than its backbone Chinese-LLaMA on legal article recommendation (0.059 vs. 0.379 accuracy), indicating possible overfitting during fine-tuning.
• General LLMs like Baichuan2 and LLaMA2 achieved comparable or better results than some legal LLMs on element recognition and judicial summarization, challenging the assumption that legal-specific models always outperform general ones.
• The benchmark revealed that only a subset of legal LLMs improve upon their general LLM backbones, and significant performance gaps remain compared to state-of-the-art models like ChatGPT.