[Paper Review] Chatlaw: A Multi-Agent Collaborative Legal Assistant with Knowledge Graph Enhanced Mixture-of-Experts Large Language Model

Jiaxi Cui, Ning, Munan+7 more|arXiv (Cornell University)|Jun 28, 2023

Topic ModelingComputer Science90 citations

TL;DR

ChatLaw presents an open-source Chinese legal LLM built on OpenLLAMA with four inference modules to reduce hallucinations, plus a keyword and law LLM for retrieval, and an Elo-based evaluation framework.

ABSTRACT

AI legal assistants based on Large Language Models (LLMs) can provide accessible legal consulting services, but the hallucination problem poses potential legal risks. This paper presents Chatlaw, an innovative legal assistant utilizing a Mixture-of-Experts (MoE) model and a multi-agent system to enhance the reliability and accuracy of AI-driven legal services. By integrating knowledge graphs with artificial screening, we construct a high-quality legal dataset to train the MoE model. This model utilizes different experts to address various legal issues, optimizing the accuracy of legal responses. Additionally, Standardized Operating Procedures (SOP), modeled after real law firm workflows, significantly reduce errors and hallucinations in legal services. Our MoE model outperforms GPT-4 in the Lawbench and Unified Qualification Exam for Legal Professionals by 7.73% in accuracy and 11 points, respectively, and also surpasses other models in multiple dimensions during real-case consultations, demonstrating our robust capability for legal consultation.

Motivation & Objective

• Motivate the need for a dedicated open-source Chinese legal LLM with high data quality and up-to-date legal knowledge.
• Develop ChatLaw by integrating domain-specific data, retrieval enhancements, and inference-time modules to mitigate hallucinations.
• Create a dataset for legal multiple-choice evaluation and establish an Elo-based model ranking for legal tasks.
• Demonstrate a multi-model controller to select appropriate models for different legal tasks.
• Release data and model components to foster open research in legal NLP.

Proposed method

• Fine-tune Ziya-LLaMA-13B using LoRA with added self-suggestion to reduce hallucinations.
• Propose a hybrid retrieval pipeline combining keyword extraction via LLM and vector-based retrieval from a law database.
• Train a Law LLM (BERT-based) on 937k case texts to extract legal provisions and interpretations for retrieval.
• Construct a Chinese legal exam dataset and an Elo-based evaluation arena to compare model performance.
• Employ a big LLM as a controller (HuggingGPT-style) to route user requests to specialized models for specific tasks.
• Release an end-to-end ChatLaw system with open-source data and components.

Experimental results

Research questions

• RQ1 Can an open-source Chinese legal LLM achieve reliable performance across multiple legal tasks (Q&A, keyword extraction, and exam-style questions) when augmented with external knowledge bases?
• RQ2 Does a hybrid retrieval approach combining keyword and vector-based methods improve legal information accuracy over pure vector retrieval?
• RQ3 How effective is a self-suggestion mechanism in reducing hallucinations during legal reasoning?
• RQ4 What is the impact of task-specific fine-tuning and model specializations on legal-domain task performance?
• RQ5 Can an Elo-based ranking framework meaningfully compare legal LLMs on standardized multiple-choice questions?

Key findings

• ChatLaw performance benefits from adding legal QA data and statute data for MCQs.
• Task-specific training significantly improves performance on targeted tasks.
• Larger parameter models tend to perform better on complex legal multiple-choice questions.
• A controller model can dynamically select the most suitable specialized model for a given user request.
• The combination of LLM-based keyword extraction, law-text similarity, and knowledge-base retrieval reduces reliance on solely vector-based retrieval.