[Paper Review] Pre-trained Language Models for the Legal Domain: A Case Study on Indian Law

Shounak Paul, Arpan Mandal+2 more|arXiv (Cornell University)|Sep 13, 2022

Artificial Intelligence in Law27 citations

TL;DR

This paper investigates continuing pre-training of two LegalBERT-family models on Indian legal text and also training a custom Indian-legal vocabulary model from scratch. It evaluates on Indian and non-Indian legal tasks, showing Indian-domain pre-training improves performance, with InLegalBERT achieving strong results across end-tasks.

ABSTRACT

NLP in the legal domain has seen increasing success with the emergence of Transformer-based Pre-trained Language Models (PLMs) pre-trained on legal text. PLMs trained over European and US legal text are available publicly; however, legal text from other domains (countries), such as India, have a lot of distinguishing characteristics. With the rapidly increasing volume of Legal NLP applications in various countries, it has become necessary to pre-train such LMs over legal text of other countries as well. In this work, we attempt to investigate pre-training in the Indian legal domain. We re-train (continue pre-training) two popular legal PLMs, LegalBERT and CaseLawBERT, on Indian legal data, as well as train a model from scratch with a vocabulary based on Indian legal text. We apply these PLMs over three benchmark legal NLP tasks -- Legal Statute Identification from facts, Semantic Segmentation of Court Judgment Documents, and Court Appeal Judgment Prediction -- over both Indian and non-Indian (EU, UK) datasets. We observe that our approach not only enhances performance on the new domain (Indian texts) but also over the original domain (European and UK texts). We also conduct explainability experiments for a qualitative comparison of all these different PLMs.

Motivation & Objective

• Assess whether continuing pre-training LegalBERT-family models on Indian legal data improves downstream legal NLP tasks.
• Develop Indian-domain PLMs (InLegalBERT, InCaseLawBERT, CustomInLawBERT) and compare to base models.
• Evaluate models on Legal Statute Identification, semantic segmentation of judgments, and Court Judgement Prediction across Indian and non-Indian datasets.
• Analyze model attentions to understand domain adaptation effects.
• Publish models publicly for reproducibility and further research.

Proposed method

• Assemble a large Indian legal corpus (~5.4M documents, ~27GB) of Supreme Court/High Court judgments and Central Acts.
• Continuing pre-training of LegalBERT and CaseLawBERT on Indian data to obtain InLegalBERT and InCaseLawBERT; train a BERT-base-uncased model from scratch with a custom Indian-legal vocabulary (CustomInLawBERT).
• Create a custom Indian vocabulary using WordPiece trained on 10% of the data for the from-scratch model.
• Use 512-token input chunks with 254-token segments to enable longer context during pre-training; employ MLM with dynamic masking and NSP with dynamic sampling.
• Evaluate perplexity on a held-out Indian-legal test set; conduct end-task fine-tuning using the HierBERT architecture (sentence/chunk encoder + LSTM with attention) across tasks.
• Compare models on Legal Statute Identification (LSI) with Indian and European datasets, Semantic Segmentation of judgments, and Court Judgement Prediction.

Experimental results

Research questions

• RQ1 Does further pre-training on Indian legal data improve performance on Indian legal NLP tasks compared to base LegalBERT/CaseLawBERT?
• RQ2 How does a from-scratch Indian-legal vocabulary model (CustomInLawBERT) compare to re-trained Indian-domain models?
• RQ3 Do Indian-domain PLMs generalize or transfer to non-Indian legal datasets (EU/UK) alongside Indian data?
• RQ4 What do attention patterns reveal about domain adaptation in Indian-specific pre-training?
• RQ5 What are the state-of-the-art performances achieved by Indian-domain PLMs on benchmark legal datasets?

Key findings

• InLegalBERT achieves the best perplexity among the studied models on Indian data (5.2547) and generally outperforms other variants on end-tasks.
• CustomInLawBERT, though trained for fewer steps (700k), improves over BERT and approaches performance of InLegalBERT/InCaseLawBERT on several datasets.
• InLegalBERT outperforms all other PLMs on most end-tasks, and establishes state-of-the-art results on 4 out of 5 datasets when compared to datasets in the original work.
• Perplexity results show Indian-domain pre-training reduces perplexity relative to base models and to EU/US-focused LegalBERT/CaseLawBERT models (e.g., BERT 25.76 vs InLegalBERT 5.25).
• Fine-tuned HierBERT architectures with encoder variants (LegalBERT, InLegalBERT, etc.) yield gains over baselines on both Indian and non-Indian benchmarks (LSI and ECtHR-B datasets).
• The authors provide publicly accessible models on HuggingFace: InLegalBERT, InCaseLawBERT, CustomInLawBERT, and accompanying pre-training code.