[Paper Review] LexGPT 0.1: pre-trained GPT-J models with Pile of Law

Jieh-Sheng Lee|arXiv (Cornell University)|Jun 5, 2023

Artificial Intelligence in Law4 citations

TL;DR

This paper introduces LexGPT 0.1, a family of GPT-J-based language models pre-trained on the Pile of Law dataset to specialize in legal text generation and classification. Fine-tuned without code modification via a 'No Code' approach, the models achieve 83.9% micro-F1 on LEDGAR and 49.6% accuracy on CaseHOLD, demonstrating feasibility for legal professionals to deploy custom classifiers with minimal technical expertise, though performance lags behind BERT-based state-of-the-art models.

ABSTRACT

This research aims to build generative language models specialized for the legal domain. The manuscript presents the development of LexGPT models based on GPT-J models and pre-trained with Pile of Law. The foundation model built in this manuscript is the initial step for the development of future applications in the legal domain, such as further training with reinforcement learning from human feedback. Another objective of this manuscript is to assist legal professionals in utilizing language models through the ``No Code'' approach. By fine-tuning models with specialized data and without modifying any source code, legal professionals can create custom language models for downstream tasks with minimum effort and technical knowledge. The downstream task in this manuscript is to turn a LexGPT model into a classifier, although the performance is notably lower than the state-of-the-art result. How to enhance downstream task performance without modifying the model or its source code is a research topic for future exploration.

Motivation & Objective

• To develop specialized generative language models for the legal domain using the Pile of Law dataset.
• To enable legal professionals to create custom language models without modifying source code, supporting a 'No Code' approach.
• To evaluate fine-tuned LexGPT models on downstream legal classification tasks using the LexGLUE benchmark.
• To explore the performance gap between zero-shot fine-tuning of GPT models and traditional BERT-based models with added classification layers.
• To identify challenges in adapting generative models to multi-label and long-sequence legal tasks.

Proposed method

• Pre-training GPT-J models (456M and 1.6B parameters) exclusively on the 256GB Pile of Law dataset, a curated legal and administrative text corpus.
• Fine-tuning the pre-trained models on downstream legal classification tasks (LEDGAR and CaseHOLD) using standard Hugging Face training pipelines without modifying model architecture.
• Applying a prompt-based classification strategy where each input is a context-prompt paired with candidate labels, and the model predicts the next token to select the correct label.
• Using a zero-shot prompting format where the model generates the label token directly from the input, avoiding architectural changes or additional classification heads.
• Evaluating performance using micro-F1 and macro-F1 scores on LEDGAR and accuracy on CaseHOLD, comparing results to existing BERT-based models.
• Excluding datasets with long sequences (ECtHR, SCOTUS) and multi-label tasks (EUR-LEX, UNFAIR-ToS) due to architectural and sequence-length constraints.

Figure 1: Training loss

Experimental results

Research questions

• RQ1 Can GPT-J models pre-trained on the Pile of Law dataset achieve competitive performance on legal text classification tasks without architectural modifications?
• RQ2 To what extent can a 'No Code' fine-tuning approach enable legal professionals to deploy custom language models with minimal technical expertise?
• RQ3 Why do fine-tuned GPT models underperform compared to BERT-based models with dedicated classification heads on legal NLU benchmarks?
• RQ4 How do sequence length and multi-label classification settings affect the performance of generative models like LexGPT in legal NLP tasks?
• RQ5 Can chain-of-thought prompting improve classification accuracy in the absence of architectural modifications?

Key findings

• The 1.6B-parameter LexGPT model achieved a micro-F1 score of 83.9% and macro-F1 score of 74.0% on the LEDGAR single-label classification task, outperforming the 456M model (83.5% micro-F1, 72.4% macro-F1).
• On the CaseHOLD multiple-choice classification task, the 456M model achieved 49.6% accuracy, significantly above random guessing (20%), but far below the 75.4% accuracy of the SOTA CaseLaw-BERT model.
• The performance gap between LexGPT and state-of-the-art BERT-based models persists despite minimal code modification, indicating limitations in prompt-based fine-tuning for classification.
• Long-sequence datasets (ECtHR, SCOTUS) and multi-label tasks (EUR-LEX, UNFAIR-ToS) were excluded due to model input length constraints and incompatibility with autoregressive generation.
• The study confirms that GPT-based models can be adapted to legal classification via prompt-based fine-tuning without code changes, validating the 'No Code' approach for legal practitioners.
• Future work is needed to explore whether chain-of-thought prompting or specialized data formatting can close the performance gap with BERT-based models.

Figure 2: Training loss