[Paper Review] Billions of Parameters Are Worth More Than In-domain Training Data: A case study in the Legal Case Entailment Task

Guilherme Moraes Rosa, Luiz Bonifacio+4 more|arXiv (Cornell University)|May 30, 2022

Topic Modeling4 citations

TL;DR

This paper demonstrates that scaling a zero-shot language model to 3 billion parameters significantly improves performance on the legal case entailment task, outperforming all fine-tuned models and ensembles in the COLIEE 2022 competition. The authors show that larger models generalize better than smaller ones, even without any in-domain fine-tuning, highlighting the value of model scale over domain-specific training data in low-resource legal NLP settings.

ABSTRACT

Recent work has shown that language models scaled to billions of parameters, such as GPT-3, perform remarkably well in zero-shot and few-shot scenarios. In this work, we experiment with zero-shot models in the legal case entailment task of the COLIEE 2022 competition. Our experiments show that scaling the number of parameters in a language model improves the F1 score of our previous zero-shot result by more than 6 points, suggesting that stronger zero-shot capability may be a characteristic of larger models, at least for this task. Our 3B-parameter zero-shot model outperforms all models, including ensembles, in the COLIEE 2021 test set and also achieves the best performance of a single model in the COLIEE 2022 competition, second only to the ensemble composed of the 3B model itself and a smaller version of the same model. Despite the challenges posed by large language models, mainly due to latency constraints in real-time applications, we provide a demonstration of our zero-shot monoT5-3b model being used in production as a search engine, including for legal documents. The code for our submission and the demo of our system are available at https://github.com/neuralmind-ai/coliee and https://neuralsearchx.neuralmind.ai, respectively.

Motivation & Objective

• To investigate whether scaling large language models improves zero-shot performance in legal case entailment without in-domain fine-tuning.
• To evaluate whether billion-parameter models generalize better than smaller models in low-resource legal NLP scenarios.
• To demonstrate the feasibility of deploying a zero-shot monoT5-3B model in production for legal document search.
• To compare zero-shot performance across different model sizes and identify the optimal architecture for legal entailment.
• To assess the impact of answer selection heuristics on final prediction quality in zero-shot settings.

Proposed method

• The authors use a zero-shot T5-based model (monoT5) without any fine-tuning on the legal case entailment dataset.
• They experiment with three model sizes: base (220M), large (770M), and 3B-parameter (monoT5-3b), all in a zero-shot setup.
• A confidence-based answer selection method is applied, combining predictions from multiple models using a weighted scoring rule with hyperparameters α, β, γ.
• The method concatenates answers from two models, removes duplicates, and selects the highest-scoring answers based on confidence thresholds.
• Ensemble predictions are generated by combining monoT5-base and monoT5-3b outputs, improving robustness and performance.
• The system is deployed in production as a legal search engine, demonstrating real-world viability despite latency constraints.

Experimental results

Research questions

• RQ1 Does scaling a language model to billions of parameters improve zero-shot performance on legal case entailment compared to smaller models?
• RQ2 Can a zero-shot model outperform fine-tuned models in the legal domain, especially when labeled data is scarce?
• RQ3 How does model size affect generalization in zero-shot legal NLP tasks, particularly in low-resource settings?
• RQ4 To what extent do heuristic answer selection methods improve zero-shot prediction quality?
• RQ5 Can a large-scale zero-shot model be effectively deployed in real-world legal search applications?

Key findings

• The monoT5-3b-zero-shot model achieved an F1 score of 0.7477 on the COLIEE 2021 test set, outperforming all single models and ensembles in the 2022 competition.
• The 3B-parameter model improved zero-shot F1 by over 6 points compared to the authors' previous zero-shot result, confirming the benefit of scale.
• The ensemble of monoT5-base and monoT5-3b achieved the highest F1 score of 0.6783 on the 2022 test set, ranking first among all submissions.
• The answer selection method improved F1 by at least one point across all models, with the best improvement on the 3B model (from 0.7373 to 0.7477).
• Despite the performance gains, the 3B model outperformed the larger T5-large model, suggesting non-monotonic performance with size in some configurations.
• The zero-shot monoT5-3b model was successfully deployed in production as a legal search engine, validating real-world applicability.