[Paper Review] Classify or Select: Neural Architectures for Extractive Document Summarization

Ramesh Nallapati, Bowen Zhou+1 more|arXiv (Cornell University)|Nov 14, 2016

Topic ModelingComputer Science17 references82 citations

TL;DR

This paper proposes two novel RNN-based neural architectures—Classifier and Selector—for extractive document summarization. The Classifier sequentially evaluates sentences in original order for inclusion, while the Selector dynamically selects salient sentences in any order, jointly modeling salience, redundancy, and content. Both models achieve state-of-the-art performance on DUC 2002 and Daily Mail datasets, with the Selector outperforming the Classifier when document structure is disrupted.

ABSTRACT

We present two novel and contrasting Recurrent Neural Network (RNN) based architectures for extractive summarization of documents. The Classifier based architecture sequentially accepts or rejects each sentence in the original document order for its membership in the final summary. The Selector architecture, on the other hand, is free to pick one sentence at a time in any arbitrary order to piece together the summary. Our models under both architectures jointly capture the notions of salience and redundancy of sentences. In addition, these models have the advantage of being very interpretable, since they allow visualization of their predictions broken up by abstract features such as information content, salience and redundancy. We show that our models reach or outperform state-of-the-art supervised models on two different corpora. We also recommend the conditions under which one architecture is superior to the other based on experimental evidence.

Motivation & Objective

• To develop interpretable neural architectures for extractive document summarization that explicitly model salience, redundancy, and content richness.
• To compare two distinct neural strategies—sequential classification vs. dynamic selection—for sentence selection in summarization.
• To evaluate the impact of document structure on model performance and identify conditions under which each architecture excels.
• To provide a framework for visualizing model decisions based on abstract features like salience and redundancy.
• To improve performance on extractive summarization while maintaining interpretability through explicit feature modeling.

Proposed method

• Use bidirectional GRUs to encode each sentence into a dense representation, followed by a second bidirectional GRU over sentence indices to capture document-level context.
• Compute a composite score for each sentence using weighted combinations of salience (cosine similarity with document representation), redundancy (cosine similarity with summary representation), content richness, and positional importance.
• Employ a binary classification head in the Classifier architecture to predict sentence inclusion in the summary in original document order.
• Implement a generative, autoregressive decoder in the Selector architecture to predict sentence indices one at a time, allowing non-sequential selection.
• Integrate a dynamic summary representation that evolves during inference to reflect the current state of the summary and penalize redundant selections.
• Use a learnable weighted score function: score = w_c·σ(W_c^T·h_j) + w_s·cos(h_j,d) + w_p·σ(W_p^T·p_j) - w_r·σ(cos(h_j,s_j)) + b, where h_j is sentence representation, d is document representation, s_j is summary representation, and p_j is positional embedding.

Experimental results

Research questions

• RQ1 How do the Classifier and Selector architectures compare in performance on structured and unstructured documents?
• RQ2 Does the ability to select sentences in any order (Selector) provide a performance advantage over sequential selection (Classifier) when document discourse structure is absent?
• RQ3 To what extent can salience, redundancy, and content richness be explicitly modeled and visualized to improve interpretability of summarization decisions?
• RQ4 Can neural models jointly optimize for salience and redundancy while maintaining high ROUGE scores on standard benchmarks?
• RQ5 How robust are these models to domain shift, particularly when fine-tuned on out-of-domain data like DUC 2002 after training on Daily Mail?

Key findings

• The Deep-Classifier model achieves a ROUGE-L score of 43.1 ± 0.9 on the Daily Mail dataset, statistically indistinguishable from Cheng & Lapata (2016) at 95% confidence interval.
• On the DUC 2002 dataset, the Deep-Classifier achieves a ROUGE-L score of 42.3, outperforming baseline models like Lead-3 (40.2) and LReg (40.3).
• When document sentence order is randomly shuffled, the Selector architecture outperforms the Classifier architecture, achieving a ROUGE-L of 32.5 vs. 32.9, indicating superior robustness to structural disruption.
• The Selector model achieves a ROUGE-L of 33.5 on shuffled data, while the Classifier model drops to 32.9, confirming that structure-agnostic selection is more effective when discourse order is lost.
• Qualitative analysis shows that the models can be interpreted via feature-wise attention, with learned weights revealing the relative importance of salience, redundancy, and position.
• The models demonstrate strong performance on both structured (e.g., news articles) and unstructured (e.g., tweet clusters) data, with the Selector architecture being particularly effective in low-structure settings.