[Paper Review] Discourse-Aware Neural Extractive Text Summarization

Jiacheng Xu, Zhe Gan+2 more|arXiv (Cornell University)|Oct 30, 2019

Topic Modeling50 references20 citations

TL;DR

This paper proposes DiscoBert, a discourse-aware extractive text summarization model that uses Elementary Discourse Units (EDUs) instead of full sentences as selection units to reduce redundancy and improve summary informativeness. By constructing RST and coreference discourse graphs and encoding them with Graph Convolutional Networks, DiscoBert captures long-range dependencies, achieving state-of-the-art performance on CNN/DM and XSum benchmarks compared to BERT-base models.

ABSTRACT

Recently BERT has been adopted for document encoding in state-of-the-art text summarization models. However, sentence-based extractive models often result in redundant or uninformative phrases in the extracted summaries. Also, long-range dependencies throughout a document are not well captured by BERT, which is pre-trained on sentence pairs instead of documents. To address these issues, we present a discourse-aware neural summarization model - DiscoBert. DiscoBert extracts sub-sentential discourse units (instead of sentences) as candidates for extractive selection on a finer granularity. To capture the long-range dependencies among discourse units, structural discourse graphs are constructed based on RST trees and coreference mentions, encoded with Graph Convolutional Networks. Experiments show that the proposed model outperforms state-of-the-art methods by a significant margin on popular summarization benchmarks compared to other BERT-base models.

Motivation & Objective

• To address redundancy and lack of long-range dependency modeling in sentence-level extractive summarization.
• To improve factual consistency and conciseness by operating at the sub-sentential discourse unit level (EDUs) instead of full sentences.
• To leverage discourse structure—specifically Rhetorical Structure Theory (RST) and coreference relations—as inductive bias for better document-level context modeling.
• To enhance BERT-based summarization by integrating discourse graphs that capture long-range interactions among discourse units.
• To achieve state-of-the-art performance on standard text summarization benchmarks using an end-to-end neural framework.

Proposed method

• The model uses BERT to encode discourse units (EDUs) derived from RST-based segmentation, replacing sentence-level encoding.
• Two discourse graphs are constructed: (1) an RST Graph from RST parse trees of EDUs, and (2) a Coreference Graph linking mentions of the same entity across the document.
• Graph Convolutional Networks (GCNs) are applied to both discourse graphs to propagate representations and model long-range dependencies among EDUs.
• The model performs extractive summarization by selecting top-ranked EDUs based on their encoded representations, with compression naturally achieved by sub-sentence selection.
• The framework is fine-tuned end-to-end on summarization datasets using standard BERT-based optimization.
• The discourse graphs are built using pre-trained discourse parsers and coreference resolution tools, with edges defined by rhetorical relations and coreference chains.

Experimental results

Research questions

• RQ1 Can sub-sentential discourse units (EDUs) serve as a more effective selection basis than full sentences for extractive summarization?
• RQ2 To what extent can discourse graphs (RST and coreference) improve long-range dependency modeling in BERT-based summarization?
• RQ3 Does integrating discourse structure via GCNs lead to more concise, less redundant, and more informative summaries?
• RQ4 How does the proposed model compare to state-of-the-art BERT-based extractive models on standard benchmarks?
• RQ5 What are the main failure modes of the discourse graph construction, and how do they affect summarization quality?

Key findings

• DiscoBert achieves new state-of-the-art results on the CNN/DM and XSum summarization benchmarks, outperforming other BERT-base models.
• The use of EDUs as selection units leads to more concise summaries with reduced redundancy compared to sentence-level extraction.
• The integration of discourse graphs significantly improves performance, demonstrating the value of structured discourse knowledge as inductive bias.
• Error analysis reveals that performance degradation is largely due to errors in RST dependency parsing and coreference resolution, particularly in anaphora resolution.
• Common errors include incorrect punctuation, missing quotation marks, and ambiguous pronouns (e.g., 'he' without clear antecedents), which affect coherence.
• The model shows consistent gains across both news and long-form document summarization tasks, indicating robustness to domain variation.