[Paper Review] Lawyer LLaMA Technical Report

Quzhe Huang, Mingxu Tao+6 more|arXiv (Cornell University)|May 24, 2023

Artificial Intelligence in Law37 citations

TL;DR

Lawyer LLaMA adapts LLaMA for the Chinese legal domain via continual pre-training on legal data, expert-driven supervised fine-tuning, and a retrieval module to fetch relevant legal articles, reducing hallucinations and improving domain reasoning.

ABSTRACT

Large Language Models (LLMs), like LLaMA, have exhibited remarkable performance across various tasks. Nevertheless, when deployed to specific domains such as law or medicine, the models still confront the challenge of a deficiency in domain-specific knowledge and an inadequate capability to leverage that knowledge to resolve domain-related problems. In this paper, we propose a new framework to adapt LLMs to specific domains and build Lawyer LLaMA, a legal domain LLM, based on this framework. Specifically, we inject domain knowledge during the continual training stage and teach the model to learn professional skills using properly designed supervised fine-tuning tasks. Moreover, to alleviate the hallucination problem during the model's generation, we add a retrieval module and extract relevant legal articles before the model answers any queries. When learning domain-specific skills, we find that experts' experience is much more useful than experiences distilled from ChatGPT, where hundreds of expert-written data outperform tens of thousands of ChatGPT-generated ones. We will release our model and data.

Motivation & Objective

• Motivate adapting LLMs to domain-specific knowledge, specifically law, to overcome generic training limitations.
• Propose a three-stage framework: inject domain knowledge, learn professional domain skills via expert-backed supervised fine-tuning, and augment with a retrieval module to ground answers.
• Demonstrate that expert-crafted data can outperform ChatGPT-generated data for domain-specific skill learning.
• Show that retrieval-based grounding reduces legal-hallucinations and improves reliability of responses.

Proposed method

• Inject domain knowledge by continual pre-training on a Chinese legal corpus plus general-domain text to prevent forgetting.
• Learn professional legal skills through supervised fine-tuning on expert-written Judicial Examination data and legally grounded tasks.
• Train with instruction-following data to improve general capability, and then fine-tune on legal- domain tasks (Judicial Examination variants, legal consultations).
• Augment responses with a retrieval module that fetches up to three relevant legal articles and train the model to filter irrelevant information.
• During training, expose the model to irrelevant articles to teach discrimination between useful and noisy inputs.
• Evaluate with automatic metrics on legal-domain tasks (Charge Prediction, JE-M) and general-domain benchmarks (C3, CMNLI, SciQ, PIQA).
• Assess hallucination types related to legal articles and measure the impact of retrieval on reducing such hallucinations.

Experimental results

Research questions

• RQ1 Can domain-specific pretraining and expert-crafted SFT improve a Chinese LLM's performance on legal tasks?
• RQ2 Does a retrieval-grounded approach reduce hallucinations and improve reliability in legal responses?
• RQ3 What type and scale of SFT data (expert-written vs. ChatGPT-generated) best teach domain-specific knowledge and reasoning?
• RQ4 Does injecting domain knowledge affect general NLP tasks beyond the legal domain?

Key findings

• Continual pre-training on a Chinese legal corpus yields large gains on legal tasks (e.g., CP and JE-M) compared to the baseline, with the biggest improvement from s1 over s0.
• Expert-written Judicial Examination data (JE-Expert) outperform ChatGPT-generated data for learning professional legal skills, even when ChatGPT-distilled datasets are much larger.
• Retrieval-augmented inputs significantly reduce hallucinations about legal articles compared with non-retrieval baselines, though retrieval is imperfect and can still introduce noise.
• Adding irrelevant articles during training helps the model learn to filter out non-useful retrieved content, reducing redundancy and off-topic responses.
• Injecting legal knowledge does not harm general-domain task performance and can even improve reasoning-related NLP tasks like CMNLI when combined with targeted fine-tuning.
• Model variants show notable improvements across both domain-specific (CP, JE-M) and several general benchmarks when trained with domain knowledge and SFT data.