[Paper Review] DISC-LawLLM: Fine-tuning Large Language Models for Intelligent Legal Services

Shengbin Yue, Wei Chen+9 more|arXiv (Cornell University)|Sep 20, 2023

Artificial Intelligence in Law24 citations

TL;DR

DISC-LawLLM fine-tunes a Chinese-domain 13B base LLM with legal syllogism prompts and retrieval augmentation, enabling legal reasoning and access to external knowledge, evaluated with a dedicated DISC-Law-Eval benchmark.

ABSTRACT

We propose DISC-LawLLM, an intelligent legal system utilizing large language models (LLMs) to provide a wide range of legal services. We adopt legal syllogism prompting strategies to construct supervised fine-tuning datasets in the Chinese Judicial domain and fine-tune LLMs with legal reasoning capability. We augment LLMs with a retrieval module to enhance models' ability to access and utilize external legal knowledge. A comprehensive legal benchmark, DISC-Law-Eval, is presented to evaluate intelligent legal systems from both objective and subjective dimensions. Quantitative and qualitative results on DISC-Law-Eval demonstrate the effectiveness of our system in serving various users across diverse legal scenarios. The detailed resources are available at https://github.com/FudanDISC/DISC-LawLLM.

Motivation & Objective

• Motivate building intelligent legal systems capable of broad legal services beyond single tasks.
• Develop DISC-Law-SFT datasets using legal syllogism prompting to instill legal reasoning in LLMs.
• Incorporate a retrieval module to access up-to-date legal knowledge and reduce hallucinations.
• Propose a comprehensive DISC-Law-Eval benchmark for objective and subjective evaluation of legal AI systems.
• Demonstrate that DISC-LawLLM outperforms existing legal LLMs on the benchmark.

Proposed method

• Construct DISC-Law-SFT datasets with two subsets to encode legal reasoning and knowledge retrieval capabilities.
• Fine-tune Baichuan-13B-Base on DISC-Law-SFT using supervised fine-tuning with specified hyperparameters.
• Introduce a retrieval augmentation module over a knowledge base of Chinese laws to provide references during generation.
• Use DISC-Law-SFT-Triplet to train the model for retrieval-aware reasoning.
• Develop DISC-Law-Eval with objective (multiple-choice across difficulty levels) and subjective (Q&A with a GPT-3.5 referee) components.

Experimental results

Research questions

• RQ1 Can DISC-LawLLM perform robust legal reasoning in Chinese following legal syllogism structures?
• RQ2 Does retrieval augmentation improve the reliability and factual grounding of legal responses?
• RQ3 How does DISC-LawLLM compare to existing general and legal LLMs on objective knowledge and subjective quality metrics?
• RQ4 Is the DISC-Law-Eval benchmark effective for assessing diverse legal services across professionals, the public, and students?

Key findings

• DISC-LawLLM with retrieval augmentation significantly outperforms competing LLMs on objective evaluation across multiple legal subjects and difficulty levels.
• Compared to GPT-3.5-turbo (175B) DISC-LawLLM shows superior performance on most objective subjects, with notable gains on hard NJE and PAE items.
• Subjective evaluation indicates DISC-LawLLM achieves higher accuracy, completeness, and clarity scores than several legal LLMs and general LLMs.
• The model demonstrates strong jurisprudential reasoning via legal syllogism prompting and enhanced reliability through retrieval-based references.
• The approach yields effective legal professional tools, consultations, and examination assistance in practical scenarios.