[Paper Review] Extractive Summarization of Legal Decisions using Multi-task Learning and Maximal Marginal Relevance

Abhishek Agarwal, Shanshan Xu+1 more|arXiv (Cornell University)|Oct 22, 2022

Topic Modeling4 citations

TL;DR

This paper proposes a multi-task learning approach for extractive summarization of legal decisions using BERT-based models and Maximal Marginal Relevance (MMR) to reduce redundancy. By jointly training on summarization and rhetorical role identification, the model achieves ROUGE scores comparable to human expert summaries, demonstrating strong performance in a low-resource legal NLP setting.

ABSTRACT

Summarizing legal decisions requires the expertise of law practitioners, which is both time- and cost-intensive. This paper presents techniques for extractive summarization of legal decisions in a low-resource setting using limited expert annotated data. We test a set of models that locate relevant content using a sequential model and tackle redundancy by leveraging maximal marginal relevance to compose summaries. We also demonstrate an implicit approach to help train our proposed models generate more informative summaries. Our multi-task learning model variant leverages rhetorical role identification as an auxiliary task to further improve the summarizer. We perform extensive experiments on datasets containing legal decisions from the US Board of Veterans' Appeals and conduct quantitative and expert-ranked evaluations of our models. Our results show that the proposed approaches can achieve ROUGE scores vis-à-vis expert extracted summaries that match those achieved by inter-annotator comparison.

Motivation & Objective

• To develop an extractive summarization system for legal decisions that performs well in low-resource settings with limited expert-annotated data.
• To reduce redundancy in legal summaries using Maximal Marginal Relevance (MMR) while preserving salient content.
• To improve summarization performance by leveraging rhetorical role identification as an auxiliary task in a multi-task learning framework.
• To evaluate the model both quantitatively using ROUGE metrics and qualitatively through expert assessment.
• To validate the model's effectiveness on real-world legal decisions from the U.S. Board of Veterans’ Appeals, focusing on PTSD compensation cases.

Proposed method

• Fine-tune a domain-specific BERT model on legal text to better capture legal terminology and sentence embeddings.
• Train a sequence labeling model to identify rhetorical roles (e.g., reasoning, evidence) in legal decisions as an auxiliary task.
• Use MMR to re-rank candidate sentences by balancing relevance and diversity, minimizing redundancy in the final summary.
• Apply multi-task learning to jointly optimize for extractive summarization and rhetorical role classification, sharing representations across tasks.
• Integrate a redundancy loss term during training to implicitly reduce semantic repetition in selected sentences.
• Combine model predictions with MMR-based selection to generate concise, informative, and non-redundant summaries.

Experimental results

Research questions

• RQ1 Can multi-task learning with rhetorical role identification improve extractive summarization performance on low-resource legal datasets?
• RQ2 How does MMR-based sentence selection compare to other redundancy reduction strategies in legal summarization?
• RQ3 To what extent do the proposed models match the quality of expert-annotated summaries in both quantitative and qualitative evaluations?
• RQ4 Does the integration of domain-specific pre-trained BERT improve summarization performance compared to general-domain models?
• RQ5 Why do qualitative expert evaluations sometimes favor single-task models despite better quantitative scores for multi-task models?

Key findings

• The multi-task learning model achieved ROUGE scores that matched or exceeded those of inter-annotator agreement, indicating performance on par with human experts.
• The inclusion of rhetorical role identification as an auxiliary task significantly improved summarization performance compared to single-task baselines.
• The MMR-based selection mechanism effectively reduced redundancy while maintaining high relevance to the source content.
• Qualitative evaluation by a legal expert confirmed that summaries generated by the proposed model were at least as good as those produced by human annotators.
• Despite higher quantitative scores, the qualitative preference for single-task models suggests potential trade-offs between ROUGE metrics and human judgment in legal summarization.
• The model generalizes well to the BVA dataset of PTSD compensation cases, though performance may vary across more diverse legal domains or fact patterns.