[Paper Review] LeXFiles and LegalLAMA: Facilitating English Multinational Legal Language Model Development

Ilias Chalkidis, Nicolas Garneau+3 more|arXiv (Cornell University)|May 12, 2023

Artificial Intelligence in Law4 citations

TL;DR

This paper introduces LeXFiles, a diverse multinational English legal corpus of 19 billion tokens spanning six legal systems (EU, UK, US, Canada, India, CoE), and LegalLAMA, a probing benchmark for assessing legal knowledge in pre-trained language models. The authors release two new legal PLMs (LexLMs) fine-tuned on LeXFiles and demonstrate that upstream masked language modeling performance and probing performance on legal concepts strongly correlate with downstream performance on LexGLUE, highlighting the importance of both domain-specific pre-training and acquired legal knowledge for effective legal NLP models.

ABSTRACT

In this work, we conduct a detailed analysis on the performance of legal-oriented pre-trained language models (PLMs). We examine the interplay between their original objective, acquired knowledge, and legal language understanding capacities which we define as the upstream, probing, and downstream performance, respectively. We consider not only the models' size but also the pre-training corpora used as important dimensions in our study. To this end, we release a multinational English legal corpus (LeXFiles) and a legal knowledge probing benchmark (LegalLAMA) to facilitate training and detailed analysis of legal-oriented PLMs. We release two new legal PLMs trained on LeXFiles and evaluate them alongside others on LegalLAMA and LexGLUE. We find that probing performance strongly correlates with upstream performance in related legal topics. On the other hand, downstream performance is mainly driven by the model's size and prior legal knowledge which can be estimated by upstream and probing performance. Based on these findings, we can conclude that both dimensions are important for those seeking the development of domain-specific PLMs.

Motivation & Objective

• To address the lack of comprehensive evaluation of legal PLMs across diverse legal systems and sub-corpora.
• To investigate the interplay between upstream pre-training objectives, probing performance on legal concepts, and downstream task performance.
• To develop and release a large-scale, multinational English legal corpus (LeXFiles) and a legal knowledge probing benchmark (LegalLAMA) to support future research.
• To train and evaluate new legal PLMs (LexLMs) on LeXFiles and compare their performance with existing models on LegalLAMA and LexGLUE.
• To determine the relative contributions of model size, pre-training corpus diversity, and legal knowledge acquisition to downstream legal NLP performance.

Proposed method

• The authors constructed LeXFiles, a multilingual legal corpus comprising 5.8 million documents from 11 sub-corpora across six English-speaking legal systems, totaling ~19 billion tokens.
• They released LegalLAMA, a probing benchmark with 8 sub-tasks assessing model knowledge of legal terminology, case law, and statutory provisions across different legal systems and topics.
• Two new legal PLMs, LexLMs, were trained by further pre-training RoBERTa-base and RoBERTa-large models on LeXFiles for 1 million additional steps.
• The authors evaluated 7 PLMs on LeXFiles sub-corpora and LegalLAMA tasks, measuring upstream (MLM), probing (knowledge acquisition), and downstream (LexGLUE) performance.
• They analyzed the correlation between upstream, probing, and downstream performance to assess the predictive power of upstream and probing scores for downstream success.
• Fine-tuning was applied to selected LexGLUE tasks to evaluate transfer performance and validate the role of pre-training in downstream generalization.

Experimental results

Research questions

• RQ1 How does the performance of legal PLMs vary across different legal systems and sub-corpora within the LeXFiles corpus?
• RQ2 To what extent does upstream masked language modeling performance predict downstream performance on legal NLP tasks?
• RQ3 How well do legal PLMs generalize to legal knowledge probing tasks, and what does this reveal about their acquired legal knowledge?
• RQ4 What is the relative contribution of model size versus pre-training corpus diversity to downstream performance?
• RQ5 Can upstream and probing performance serve as reliable proxies for predicting downstream performance in legal NLP?

Key findings

• Upstream masked language modeling performance strongly correlates with probing performance on legal concepts, indicating that models pre-trained on diverse legal corpora acquire systematic legal knowledge.
• Downstream performance on LexGLUE is primarily driven by model size and prior legal knowledge, which can be estimated via upstream and probing performance.
• The LeXFiles corpus contains 19 billion tokens across 11 sub-corpora, with U.S. case law (59.2% of tokens) and U.S. contracts (27.3%) being the largest components, reflecting a significant U.S. bias in existing legal corpora.
• LegalLAMA includes 8 probing sub-tasks covering legal terminology, case law, and statutory provisions, with masked terms drawn from key legal instruments such as the ECHR and U.S. criminal codes.
• Models fine-tuned on LexGLUE tasks show that upstream and probing performance are strong predictors of downstream success, suggesting that knowledge acquisition during pre-training is a key determinant of transfer performance.
• The release of LexLMs, pre-trained on LeXFiles, provides a new benchmark for training and evaluating legal PLMs with improved coverage of multinational legal systems.