[Paper Review] THUIR@COLIEE-2020: Leveraging Semantic Understanding and Exact Matching for Legal Case Retrieval and Entailment

Yunqiu Shao, Bulou Liu+4 more|arXiv (Cornell University)|Dec 24, 2020

Artificial Intelligence in Law10 references8 citations

TL;DR

This paper presents a hybrid approach combining neural semantic understanding (BERT-PLI) and exact matching (word-entity duet) for legal case retrieval and entailment in the COLIEE-2020 competition. The method leverages learning-to-rank to integrate both strategies, achieving 2nd place in retrieval and 4th in entailment, demonstrating that semantic models outperform exact matching in entailment while both complement each other in retrieval.

ABSTRACT

In this paper, we present our methodologies for tackling the challenges of legal case retrieval and entailment in the Competition on Legal Information Extraction / Entailment 2020 (COLIEE-2020). We participated in the two case law tasks, i.e., the legal case retrieval task and the legal case entailment task. Task 1 (the retrieval task) aims to automatically identify supporting cases from the case law corpus given a new case, and Task 2 (the entailment task) to identify specific paragraphs that entail the decision of a new case in a relevant case. In both tasks, we employed the neural models for semantic understanding and the traditional retrieval models for exact matching. As a result, our team (TLIR) ranked 2nd among all of the teams in Task 1 and 3rd among teams in Task 2. Experimental results suggest that combing models of semantic understanding and exact matching benefits the legal case retrieval task while the legal case entailment task relies more on semantic understanding.

Motivation & Objective

• To address the challenges of legal case retrieval and entailment in the COLIEE-2020 competition using both semantic understanding and exact matching techniques.
• To investigate whether combining semantic and lexical matching methods improves performance in legal information retrieval tasks.
• To evaluate the effectiveness of asymmetric truncation in BERT fine-tuning for legal entailment, where query fragments are critical.
• To analyze the limitations of model combination in low-data regimes and identify key factors affecting performance in legal NLP tasks.

Proposed method

• Employed the word-entity duet framework for exact matching to capture lexical and entity-level similarities in legal cases.
• Applied the BERT-PLI model for deep semantic understanding by modeling paragraph-level interactions in long legal documents.
• Combined features from both models using RankSVM for learning-to-rank in the retrieval task.
• Used asymmetric truncation during BERT fine-tuning in the entailment task to preserve information in the decision fragment (query).
• Integrated additional meta-features and exact matching features with BERT representations in the entailment task.
• Conducted experiments with three distinct runs per task, varying model combinations and fine-tuning strategies.

Experimental results

Research questions

• RQ1 How do semantic understanding models and exact matching techniques compare in legal case retrieval and entailment tasks?
• RQ2 Can combining semantic and lexical matching improve retrieval performance in legal case retrieval?
• RQ3 Does asymmetric truncation during BERT fine-tuning enhance performance in legal case entailment?
• RQ4 Why does model combination fail to improve performance in the entailment task despite success in retrieval?
• RQ5 To what extent does data scarcity and dataset bias affect model generalization in legal NLP tasks?

Key findings

• The BERT-PLI model significantly outperformed the word-entity duet model in legal case retrieval, indicating that semantic understanding is more effective than exact matching in this task.
• Combining semantic and exact matching models via learning-to-rank improved performance over either method alone, confirming their complementary nature in retrieval.
• The best run in Task 1 (retrieval) achieved 2nd place with an F1 score close to the top-ranked team, differing by less than 0.01 F1 points.
• In Task 2 (entailment), the best run using asymmetric truncation ranked 4th, indicating that preserving query fragment information enhances performance.
• Model combination did not improve results in Task 2, likely due to the task's reliance on deep semantic understanding and the risk of overfitting in small datasets.
• The top-performing team in Task 2 (JNLP-BMWT) significantly outperformed all others, suggesting room for improvement in current approaches.