[Paper Review] BASS: Boosting Abstractive Summarization with Unified Semantic Graph

Wenhao Wu, Wei Li+6 more|arXiv (Cornell University)|May 25, 2021

Topic Modeling52 references4 citations

TL;DR

This paper proposes BASS, a graph-based framework that enhances abstractive summarization for long and multi-document inputs by constructing a unified semantic graph to model long-distance dependencies and global structure. By integrating coreference resolution and phrase-level relations into a graph-encoder and graph-propagation attention in the decoder, BASS significantly improves content selection and coherence, outperforming strong baselines on both BIGPATENT and WikiSUM datasets with state-of-the-art ROUGE scores and human evaluation results.

ABSTRACT

Abstractive summarization for long-document or multi-document remains challenging for the Seq2Seq architecture, as Seq2Seq is not good at analyzing long-distance relations in text. In this paper, we present BASS, a novel framework for Boosting Abstractive Summarization based on a unified Semantic graph, which aggregates co-referent phrases distributing across a long range of context and conveys rich relations between phrases. Further, a graph-based encoder-decoder model is proposed to improve both the document representation and summary generation process by leveraging the graph structure. Specifically, several graph augmentation methods are designed to encode both the explicit and implicit relations in the text while the graph-propagation attention mechanism is developed in the decoder to select salient content into the summary. Empirical results show that the proposed architecture brings substantial improvements for both long-document and multi-document summarization tasks.

Motivation & Objective

• To address the limitations of Seq2Seq models in handling long-document and multi-document summarization due to poor long-distance relation modeling.
• To improve document representation and summary generation by leveraging a unified semantic graph that captures global structure and co-reference relations.
• To enhance content selection and coherence in abstractive summarization through graph-based encoding and decoding mechanisms.
• To validate the effectiveness of the unified semantic graph in complex summarization scenarios with automatic and human evaluations.

Proposed method

• Construct a phrase-level unified semantic graph using dependency parsing and coreference resolution to aggregate co-referent phrases across long contexts.
• Apply graph augmentation techniques to encode both explicit and implicit semantic relations between phrases.
• Design a graph-encoder that leverages the semantic graph to improve document representation by modeling inter-phrase relations.
• Introduce a graph-propagation attention mechanism in the decoder to guide summary generation by selecting salient, graph-connected content.
• Integrate the graph structure into both encoder and decoder to enhance global context understanding and coherence in generated summaries.
• Utilize pre-trained language models as the backbone while enhancing them with graph-based inductive biases for better summarization.

Experimental results

Research questions

• RQ1 Can a unified semantic graph effectively model long-distance dependencies and global structure in long and multi-document inputs for abstractive summarization?
• RQ2 How does graph-based encoding improve document representation compared to standard Seq2Seq or sentence-level graph models?
• RQ3 To what extent does graph-propagation attention in the decoder enhance salient content selection and summary coherence?
• RQ4 Does the proposed framework outperform strong baselines in both automatic and human evaluations on long-document and multi-document summarization?
• RQ5 How does the model maintain performance as input length increases, especially beyond the context limits of standard models?

Key findings

• BASS achieves state-of-the-art performance on the BIGPATENT long-document summarization dataset, with ROUGE-1 scores significantly improving at input lengths up to 3000 tokens.
• On the WikiSUM multi-document summarization dataset, BASS outperforms strong baselines including RoBERTaS2S and GraphSum in both automatic and human evaluations.
• Human evaluation shows BASS generates more fluent, concise, and informative summaries, with a rating of 1.18 compared to 0.58 for GraphSum and 0.48 for RoBERTaS2S.
• BASS generates more abstract summaries than previous models, with a higher average number of novel n-grams, indicating stronger abstractive capabilities.
• The model maintains performance gains even at extreme input lengths, unlike GraphSum whose performance degrades beyond 3000 tokens.
• The graph-propagation attention mechanism effectively guides content selection, reducing hallucination and improving coherence in generated summaries.