[Paper Review] Hierarchical Transformers for Multi-Document Summarization

Yang Liu, Mirella Lapata|arXiv (Cornell University)|May 30, 2019

Topic Modeling39 references34 citations

TL;DR

The paper introduces a Hierarchical Transformer model that encodes multiple input documents with local and global attention, optionally incorporating graph-informed attention, to produce abstractive multi-document summaries on WikiSum, outperforming strong baselines.

ABSTRACT

In this paper, we develop a neural summarization model which can effectively process multiple input documents and distill Transformer architecture with the ability to encode documents in a hierarchical manner. We represent cross-document relationships via an attention mechanism which allows to share information as opposed to simply concatenating text spans and processing them as a flat sequence. Our model learns latent dependencies among textual units, but can also take advantage of explicit graph representations focusing on similarity or discourse relations. Empirical results on the WikiSum dataset demonstrate that the proposed architecture brings substantial improvements over several strong baselines.

Motivation & Objective

• Develop a neural model capable of generating abstractive summaries from clusters of input documents.
• Address challenges of multi-document inputs by capturing cross-document relations beyond flat concatenation.
• Leverage hierarchical encoding and graph-based signals to improve summarization quality on WikiSum.
• Evaluate whether document ranking and graph-informed attention improve performance.

Proposed method

• Extend Transformer with hierarchical encoding: local paragraph-level encoders plus global inter-paragraph attention.
• Introduce a multi-head paragraph pooling to obtain fixed-length paragraph representations.
• Incorporate inter-paragraph attention to share information across documents.
• Optionally replace inter-paragraph attention head with a graph matrix (lexical or discourse-based) to inform attention.
• Rank input paragraphs with a learning-based paragraph ranking module using ROUGE-2 as supervision.
• Train with maximum likelihood; decode with beam search and length penalty.
• Compare against Lead, LexRank, Flat Transformer variants, and Transformer-based baselines (T-DMCA).
• Evaluate on WikiSum with ROUGE-F1 metrics and conduct human evaluation (QA-based and Best-Worst scaling).

Experimental results

Research questions

• RQ1 Can hierarchical encoding of multiple documents improve abstractive summarization over flat concatenation?
• RQ2 Do inter-paragraph attention and graph-informed attention improve summary quality?
• RQ3 Does learning-based paragraph ranking improve performance versus tf-idf similarity ranking?
• RQ4 What is the impact of longer input at test time on hierarchical models?
• RQ5 How do human judgments compare with automatic metrics for multi-document abstractive summarization?

Key findings

• Hierarchical Transformer (HT) outperforms strong baselines on WikiSum across ROUGE-1, ROUGE-2, and ROUGE-L.
• Incorporating longer input (around 1,600 tokens) yields better performance for HT, and longer test input (3,000 tokens) further improves results.
• Graph-informed attention (discourse graph) boosts ROUGE-L by about 0.16 points for HT.
• Learning-based paragraph ranking improves input selection, yielding higher ROUGE-L recall than tf-idf similarity ranking.
• Ablation shows paragraph position, multi-head pooling, and the global transformer layer each contribute to stronger performance.
• Human evaluation favors HT over Lead, FT, and T-DMCA, with statistically significant differences.