[Paper Review] LawGPT: A Chinese Legal Knowledge-Enhanced Large Language Model

Zhi Qiang Zhou, Jiang-Xin Shi+5 more|arXiv (Cornell University)|Jun 7, 2024

Artificial Intelligence in Law14 citations

TL;DR

LawGPT is the first open-source Chinese legal LLM enhanced with a legal-oriented pre-training corpus and a knowledge-driven fine-tuning stage, outperforming the open-source LLaMA 7B on major legal tasks. It preserves data privacy by self-hosting.

ABSTRACT

Large language models (LLMs), including both proprietary and open-source models, have showcased remarkable capabilities in addressing a wide range of downstream tasks. Nonetheless, when it comes to practical Chinese legal tasks, these models fail to meet the actual requirements. Proprietary models do not ensure data privacy for sensitive legal cases, while open-source models demonstrate unsatisfactory performance due to their lack of legal knowledge. To address this problem, we introduce LawGPT, the first open-source model specifically designed for Chinese legal applications. LawGPT comprises two key components: legal-oriented pre-training and legal supervised fine-tuning. Specifically, we employ large-scale Chinese legal documents for legal-oriented pre-training to incorporate legal domain knowledge. To further improve the model's performance on downstream legal tasks, we create a knowledge-driven instruction dataset for legal supervised fine-tuning. Our experimental results demonstrate that LawGPT outperforms the open-source LLaMA 7B model. Our code and resources are publicly available at https://github.com/pengxiao-song/LaWGPT and have received 5.7K stars on GitHub.

Motivation & Objective

• Motivate creation of an open-source Chinese legal LLM to address data privacy and knowledge gaps in existing models.
• Incorporate legal domain knowledge through large-scale legal pre-training.
• Enhance downstream legal task performance via knowledge-driven supervised fine-tuning.
• Demonstrate LawGPT’s effectiveness against open-source baselines and discuss privacy advantages.

Proposed method

• Develop a legal-oriented pre-training (LPT) using a 500K legal document corpus.
• Apply LoRA to fine-tune the base model during LPT.
• Construct a 30K-strong knowledge-driven instruction dataset for legal supervised fine-tuning (LFT).
• Fine-tune the LPT model on D_LFT using Alpaca-style templates to obtain LawGPT (LFT).
• During inference, wrap instructions with an Alpaca template and generate responses autoregressively.
• Evaluate zero-shot performance on eight legal tasks and compare with GPT-3.5 Turbo, GPT-4, and LLaMA 7B.

Experimental results

Research questions

• RQ1 How does a legal-oriented pre-training affect Chinese legal understanding and reasoning in LLMs?
• RQ2 Can knowledge-driven supervised fine-tuning improve open-source LLM performance on legal tasks?
• RQ3 How does LawGPT compare to proprietary models and open-source baselines in zero-shot legal tasks?
• RQ4 What privacy advantages are gained by deploying LawGPT as an open-source, self-hosted model?

Key findings

• LawGPT surpasses the open-source LLaMA 7B model on major legal tasks in zero-shot settings.
• LawGPT remains behind proprietary models GPT-3.5 Turbo and GPT-4 in overall performance.
• The combination of LPT and LFT improves legal task performance over base open-source models.
• LawGPT enables private, self-hosted deployment with data privacy advantages.
• The paper provides public code and resources on GitHub (5.7K stars noted).