[Paper Review] Caseformer: Pre-training for Legal Case Retrieval Based on Inter-Case Distinctions

Weihang Su, Qingyao Ai+6 more|arXiv (Cornell University)|Nov 1, 2023

Artificial Intelligence in Law4 citations

TL;DR

This paper proposes Caseformer, a novel pre-training framework for legal case retrieval that learns legal knowledge and relevance-matching patterns through three unsupervised tasks—Fact-Document Matching, Legal Judgment Prediction, and Inter-Case Contrastive Learning—achieving state-of-the-art performance in both zero-shot and fine-tuning settings on Chinese and English legal datasets without requiring human-annotated data.

ABSTRACT

Legal case retrieval aims to help legal workers find relevant cases related to their cases at hand, which is important for the guarantee of fairness and justice in legal judgments. While recent advances in neural retrieval methods have significantly improved the performance of open-domain retrieval tasks (e.g., Web search), their advantages have not been observed in legal case retrieval due to their thirst for annotated data. As annotating large-scale training data in legal domains is prohibitive due to the need for domain expertise, traditional search techniques based on lexical matching such as TF-IDF, BM25, and Query Likelihood are still prevalent in legal case retrieval systems. While previous studies have designed several pre-training methods for IR models in open-domain tasks, these methods are usually suboptimal in legal case retrieval because they cannot understand and capture the key knowledge and data structures in the legal corpus. To this end, we propose a novel pre-training framework named Caseformer that enables the pre-trained models to learn legal knowledge and domain-specific relevance information in legal case retrieval without any human-labeled data. Through three unsupervised learning tasks, Caseformer is able to capture the special language, document structure, and relevance patterns of legal case documents, making it a strong backbone for downstream legal case retrieval tasks. Experimental results show that our model has achieved state-of-the-art performance in both zero-shot and full-data fine-tuning settings. Also, experiments on both Chinese and English legal datasets demonstrate that the effectiveness of Caseformer is language-independent in legal case retrieval.

Motivation & Objective

• Address the lack of large-scale annotated training data in legal case retrieval, which limits the effectiveness of neural retrieval models.
• Overcome the limitations of general-purpose pre-trained models like BERT and RoBERTa in understanding legal document structure and legal relevance.
• Develop a pre-training framework that captures domain-specific knowledge, language patterns, and inter-case relationships in legal documents without human-annotated supervision.
• Enable strong performance in both zero-shot and fine-tuning scenarios across multiple languages, particularly Chinese and English legal corpora.

Proposed method

• Propose a three-task unsupervised pre-training framework: Fact-Document Matching (FDM), Legal Judgment Prediction (LJP), and Inter-Case Contrastive Learning (ICCL).
• Use FDM to align legal facts with corresponding case documents by predicting whether a fact description matches a case.
• Use LJP to predict legal judgments (e.g., charges or sentences) from case text, enabling the model to learn legal reasoning and similarity.
• Use ICCL to contrast pairs of cases based on their legal similarity, encouraging the model to learn fine-grained distinctions between cases.
• Design the framework to support both dense retrieval and neural re-ranking models, enhancing downstream retrieval performance.
• Train the model on large-scale legal corpora without human-annotated query-document pairs, leveraging intrinsic document structure and legal semantics.

Experimental results

Research questions

• RQ1 Can a pre-training framework learn legal knowledge and relevance-matching patterns in legal case retrieval without human-annotated data?
• RQ2 How do the individual pre-training tasks (FDM, LJP, ICCL) contribute to the model’s ability to distinguish legal cases?
• RQ3 Does the proposed framework generalize across languages, as evidenced by performance on both Chinese and English legal datasets?
• RQ4 Can the model achieve state-of-the-art performance in zero-shot retrieval without fine-tuning?
• RQ5 How does the model’s ability to capture legal similarity compare to existing models like BERT-XS, text-embedding-ada-002, and Lawformer?

Key findings

• Caseformer achieves state-of-the-art performance in both zero-shot and fine-tuning settings on multiple legal retrieval benchmarks, including Chinese and English datasets.
• The ablation study shows that removing the Legal Judgment Prediction (LJP) task causes the largest performance drop, indicating its critical role in learning legal similarity.
• The Inter-Case Contrastive Learning (ICCL) task contributes significantly to distinguishing cases with similar category charges but different legal provisions.
• t-SNE visualization confirms that Caseformer clusters cases by specific crime type, while baselines like BERT-XS and text-embedding-ada-002 only group by category charge, showing finer-grained legal understanding.
• Caseformer outperforms existing models such as Lawformer and text-embedding-ada-002 in zero-shot retrieval, demonstrating superior generalization and legal reasoning ability.
• The model’s effectiveness is language-independent, as it achieves strong performance on both Chinese and English legal datasets, indicating broad applicability.