[Paper Review] Multi-LexSum: Real-World Summaries of Civil Rights Lawsuits at Multiple Granularities

Zejiang Shen, Kyle Lo+4 more|arXiv (Cornell University)|Jun 22, 2022

Artificial Intelligence in Law20 citations

TL;DR

Multi-LexSum provides 9,280 expert-authored, multi-granularity summaries for U.S. federal civil rights lawsuits drawn from the CRLC, and analyzes how current models underperform on this challenging, long, multi-document summarization task.

ABSTRACT

With the advent of large language models, methods for abstractive summarization have made great strides, creating potential for use in applications to aid knowledge workers processing unwieldy document collections. One such setting is the Civil Rights Litigation Clearinghouse (CRLC) (https://clearinghouse.net),which posts information about large-scale civil rights lawsuits, serving lawyers, scholars, and the general public. Today, summarization in the CRLC requires extensive training of lawyers and law students who spend hours per case understanding multiple relevant documents in order to produce high-quality summaries of key events and outcomes. Motivated by this ongoing real-world summarization effort, we introduce Multi-LexSum, a collection of 9,280 expert-authored summaries drawn from ongoing CRLC writing. Multi-LexSum presents a challenging multi-document summarization task given the length of the source documents, often exceeding two hundred pages per case. Furthermore, Multi-LexSum is distinct from other datasets in its multiple target summaries, each at a different granularity (ranging from one-sentence "extreme" summaries to multi-paragraph narrations of over five hundred words). We present extensive analysis demonstrating that despite the high-quality summaries in the training data (adhering to strict content and style guidelines), state-of-the-art summarization models perform poorly on this task. We release Multi-LexSum for further research in summarization methods as well as to facilitate development of applications to assist in the CRLC's mission at https://multilexsum.github.io.

Motivation & Objective

• Motivate real-world legal summarization as a high-cost, high-value task requiring synthesis across many documents.
• Introduce Multi-LexSum, a large, expert-authored dataset with long source documents and multi-granularity summaries.
• Analyze baseline model performance and highlight gaps for abstractive, multi-document summarization in the legal domain.

Proposed method

• Assemble ~40k source documents and ~9k expert-written summaries from the CRLC.
• Provide three target granularities for each case: tiny (≈25 words), short (≈130 words), long (≈650 words).
• Evaluate state-of-the-art abstractive models (BART, PEGASUS, LED, PRIMERA) on multi-document tasks with long inputs.
• Compare abstractive models against extractive baselines and analyze performance as input length and granularity increase.
• Experiment with multi-task and multitask training setups to leverage summaries at multiple granularities.
• Conduct human evaluation with CRLC-trained law students to assess usability and accuracy of generated summaries.

Experimental results

Research questions

• RQ1 Can models generate and synthesize information from massive multi-document sources to produce long, short, and tiny summaries (d→l, d→s, d→t)?
• RQ2 Can models be configured to produce accurate and usable summaries at different granularities (l→s, l→t, s→t)?
• RQ3 Does leveraging multiple granularities via multitask or progressive summarization improve performance, especially for long summaries?
• RQ4 Do multi-granularity training paradigms reduce hallucinations and improve factual alignment with source material?

Key findings

• Existing summarization models underperform on Multi-LexSum, with human-judged quality averages around 0.43 on a 0–3 scale for model outputs.
• Long input models (LED, PRIMERA) outperform traditional short-input models on all three multi-document tasks.
• Long summaries remain challenging to replicate at human parity; even best models yield far shorter outputs than gold long summaries (average 647 words vs. ~416 words for some models).
• Training on multiple granularities (three-task multitask setup) improves long-summary performance by 11–17% in automated metrics for l→s and l→t tasks.
• Condensing long/short summaries into shorter ones (d→s, d→t, l→s, l→t, s→t) yields substantial gains, and using gold summaries as inputs further boosts performance, suggesting benefits to pipeline or multi-stage approaches.
• Human-in-the-loop evaluation reveals that, despite automated gains, end-to-end usability remains limited without human curation and adaptive system design; a CRLC-tuned system with snippet-level selection reduced errors but still required substantial edits.