[Paper Review] BRIO: Bringing Order to Abstractive Summarization

Yixin Liu, Pengfei Liu+2 more|arXiv (Cornell University)|Mar 31, 2022

Topic Modeling4 citations

TL;DR

BRIO proposes a novel training paradigm for abstractive summarization that replaces maximum likelihood estimation's deterministic target distribution with a non-deterministic one, assigning probability mass to candidate summaries based on their quality via contrastive learning. This dual-role training improves ROUGE scores to 47.78 (ROUGE-1) on CNN/DailyMail and 49.07 on XSum, while enhancing model calibration and relative ranking of summary quality.

ABSTRACT

Abstractive summarization models are commonly trained using maximum likelihood estimation, which assumes a deterministic (one-point) target distribution in which an ideal model will assign all the probability mass to the reference summary. This assumption may lead to performance degradation during inference, where the model needs to compare several system-generated (candidate) summaries that have deviated from the reference summary. To address this problem, we propose a novel training paradigm which assumes a non-deterministic distribution so that different candidate summaries are assigned probability mass according to their quality. Our method achieves a new state-of-the-art result on the CNN/DailyMail (47.78 ROUGE-1) and XSum (49.07 ROUGE-1) datasets. Further analysis also shows that our model can estimate probabilities of candidate summaries that are more correlated with their level of quality.

Motivation & Objective

• To address the limitation of maximum likelihood estimation in abstractive summarization, which treats the reference summary as the only correct output and fails to rank candidate summaries by quality.
• To improve model performance during inference by enabling accurate relative comparison of candidate summaries, reducing exposure bias effects.
• To train a single model to serve dual roles: as a generation model and as a reference-free evaluation model for candidate summaries.
• To enhance the alignment between model-predicted probabilities and actual summary quality metrics like ROUGE.
• To develop a training paradigm that better calibrates model confidence and improves robustness to noisy patterns in training data.

Proposed method

• Replace the standard maximum likelihood estimation (MLE) loss with a contrastive loss that models a non-deterministic target distribution, assigning probability mass to candidate summaries based on their quality.
• Use pre-trained abstractive models (e.g., BART) to generate diverse candidate summaries for each input document during training.
• Apply contrastive learning to encourage the model to assign higher probabilities to higher-quality candidate summaries, as measured by ROUGE scores.
• Train the model using a dual objective: standard MLE for reference summary generation and contrastive loss for relative ranking of candidate summaries.
• Integrate the contrastive loss into the training process without modifying the autoregressive generation mechanism, enabling end-to-end training.
• Extend the method to few-shot fine-tuning by sampling only 100 or 1000 examples, demonstrating effectiveness with minimal data.

Experimental results

Research questions

• RQ1 Can a single abstractive summarization model be trained to both generate summaries and accurately rank candidate summaries by quality?
• RQ2 Does replacing deterministic MLE with a non-deterministic distribution improve summary quality and model calibration?
• RQ3 To what extent does the contrastive loss improve the alignment between model-predicted probabilities and actual ROUGE scores of candidate summaries?
• RQ4 Can the proposed method generalize to few-shot fine-tuning settings with minimal training data?
• RQ5 Does the model learn to filter out low-quality or noisy patterns (e.g., 'click here' links) in training data?

Key findings

• BRIO achieves a new state-of-the-art result on CNN/DailyMail with 47.78 ROUGE-1, surpassing previous methods.
• On the XSum dataset, BRIO achieves 49.07 ROUGE-1, setting a new SOTA performance.
• The model improves relative ranking accuracy to 79.63% on CNNDM, significantly outperforming BART’s 54.80% in assigning higher probabilities to better summaries.
• BRIO-Mul learns to ignore noisy patterns like 'click here' in summaries, which BART incorrectly learns and generates.
• The model shows better calibration, especially on XSum, with reliability graphs indicating reduced overconfidence compared to baseline models.
• In few-shot fine-tuning, BRIO-Few achieves gains of +1.52 ROUGE-1 on CNNDM and +0.49 on XSum using only 100 or 1000 training examples.