[论文解读] CUAD: An Expert-Annotated NLP Dataset for Legal Contract Review

Dan Hendrycks, Collin Burns+2 more|arXiv (Cornell University)|Mar 10, 2021

Artificial Intelligence in Law参考文献 23被引用 97

一句话总结

CUAD introduces a large, expert-annotated contract review dataset with 13,101 annotations across 41 labels and evaluates transformer models, showing data size and model design influence performance in specialized legal NLP.

ABSTRACT

Many specialized domains remain untouched by deep learning, as large labeled datasets require expensive expert annotators. We address this bottleneck within the legal domain by introducing the Contract Understanding Atticus Dataset (CUAD), a new dataset for legal contract review. CUAD was created with dozens of legal experts from The Atticus Project and consists of over 13,000 annotations. The task is to highlight salient portions of a contract that are important for a human to review. We find that Transformer models have nascent performance, but that this performance is strongly influenced by model design and training dataset size. Despite these promising results, there is still substantial room for improvement. As one of the only large, specialized NLP benchmarks annotated by experts, CUAD can serve as a challenging research benchmark for the broader NLP community.

研究动机与目标

• Provide a large, expert-annotated dataset for contract review to study NLP in a highly specialized domain.
• Quantify how transformer models perform on extractive clause highlighting across 41 label categories.
• Assess the impact of training data size and model design on contract review performance.
• Demonstrate the value of domain-specific annotated data for improving NLP in law.

提出的方法

• Formulate the task as predicting start/end spans of contract text relevant to each of 41 label categories, similar to extractive QA with possible no-answer cases.
• Use a sliding window to handle long contracts and fine-tune multiple pretrained language models with HuggingFace Transformers on CUAD.
• Evaluate models using AUPR and Precision at 80%/90% recall, with Jaccard similarity to match spans.
• Downweight non-relevant (negative) features during training to address extreme class imbalance.
• Pretrain contract-domain RoBERTa-base on ~8 GB of unlabeled EDGAR contracts to test domain pretraining benefits.

实验结果

研究问题

• RQ1 Can large domain-specific annotated datasets enable NLP models to identify salient contract clauses effectively?
• RQ2 How do model design and training data size influence performance on contract review tasks?
• RQ3 What is the relative benefit of domain-specific pretraining for contract analysis?

主要发现

• DeBERTa-xlarge achieves the best overall AUPR (47.8%), and Precision@80% Recall of 44.0%, with 17.8% at Precision@90% Recall.
• Even large model size yields variable gains; DeBERTa outperforms BERT substantially, while some size increases yield modest improvements.
• Domain-specific pretraining on contracts provides a modest AUPR gain (~3 percentage points) over non-pretrained baselines.
• Performance varies significantly by label category, with some categories near ceiling and others around 20–30% AUPR.
• Training data size strongly affects performance; increasing data by an order of magnitude yields substantial gains (e.g., 27.6% to 42.6% AUPR for RoBERTa-base).
• CUAD is a challenging benchmark with NAS-like need for high recall in legal review contexts.