[論文レビュー] BERT Goes to Law School: Quantifying the Competitive Advantage of Access to Large Legal Corpora in Contract Understanding

Emad Elwany, D. Moore+1 more|arXiv (Cornell University)|Nov 1, 2019

Artificial Intelligence in Law参考文献 6被引用数 33

ひとこと要約

本論文は、巨大でドメイン特化の法的コーパス上で BERT のファインチューニングを行うと、契約条件分類でわずかながらも意味のある改善が見られ、トレーニングが高速化されることを示しており、未アノテーションの法的コーパスが競争上の優位性になることを示唆している。

ABSTRACT

Fine-tuning language models, such as BERT, on domain specific corpora has proven to be valuable in domains like scientific papers and biomedical text. In this paper, we show that fine-tuning BERT on legal documents similarly provides valuable improvements on NLP tasks in the legal domain. Demonstrating this outcome is significant for analyzing commercial agreements, because obtaining large legal corpora is challenging due to their confidential nature. As such, we show that having access to large legal corpora is a competitive advantage for commercial applications, and academic research on analyzing contracts.

研究の動機と目的

• Assess how a domain-specific legal corpus impacts BERT-based contract classification performance.
• Quantify performance gains from fine-tuning BERT on legal text versus using pre-trained BERT.
• Examine how increasing legal corpus size affects end-task accuracy and training speed.

提案手法

• Fine-tune a BERT-Base uncased model on a proprietary legal corpus extracted from hundreds of thousands of agreements.
• Compare baseline Bag-of-Words + neural classifier, BERT with and without end-task fine-tuning, and variations with unfrozen vs frozen BERT layers.
• Evaluate on a hand-annotated contract term classification task (fixed vs auto-renewing).
• Use train/validation/test splits with early stopping based on validation loss.
• Report metrics: Precision, Recall, F1, and MCC (weighted).
• Present results across configurations to assess the impact of domain-specific fine-tuning and corpus size.

実験結果

リサーチクエスチョン

• RQ1 Does pre-trained BERT outperform a baseline lexical model on legal contract classification?
• RQ2 Does further fine-tuning BERT on a large legal corpus improve performance over using a pre-trained model alone?
• RQ3 How does increasing the size of the legal corpus used for fine-tuning affect accuracy and training efficiency?
• RQ4 Is freezing BERT layers detrimental or beneficial for this end task, and how does end-task fine-tuning interact with it?

主な発見

モデル	Precision	Recall	F1	MCC
Baseline	0.848	0.846	0.845	0.689
BERT	0.898	0.895	0.894	0.789
FT BERT (SM)	0.900	0.898	0.898	0.795
FT BERT (LG)	0.904	0.903	0.901	0.799

• Pre-trained BERT improves over the baseline Bag-of-Words approach.
• Fine-tuning BERT on a legal corpus yields additional performance gains over the pre-trained model.
• Unfreezing BERT layers with fine-tuning achieves the strongest results among the tested configurations.
• Larger legal corpora for fine-tuning provide further improvements in accuracy and faster training convergence.
• A large legal corpus, even if unannotated, constitutes a valuable asset and competitive advantage for legal NLP applications.