[Paper Review] AttSum: Joint Learning of Focusing and Summarization with Neural Attention

Ziqiang Cao, Wenjie Li+3 more|arXiv (Cornell University)|Apr 1, 2016

Topic ModelingComputer Science27 references58 citations

TL;DR

AttSum proposes a joint neural network model that uses an attention mechanism to simultaneously learn sentence relevance to a query and sentence saliency, generating distributed sentence and document embeddings without hand-crafted features. Evaluated on DUC 2005–2007 benchmarks, AttSum achieves competitive performance and effectively identifies content that meets query needs, outperforming systems relying on engineered features.

ABSTRACT

Query relevance ranking and sentence saliency ranking are the two main tasks in extractive query-focused summarization. Previous supervised summarization systems often perform the two tasks in isolation. However, since reference summaries are the trade-off between relevance and saliency, using them as supervision, neither of the two rankers could be trained well. This paper proposes a novel summarization system called AttSum, which tackles the two tasks jointly. It automatically learns distributed representations for sentences as well as the document cluster. Meanwhile, it applies the attention mechanism to simulate the attentive reading of human behavior when a query is given. Extensive experiments are conducted on DUC query-focused summarization benchmark datasets. Without using any hand-crafted features, AttSum achieves competitive performance. It is also observed that the sentences recognized to focus on the query indeed meet the query need.

Motivation & Objective

• To address the limitation of isolated training of query relevance and saliency ranking in extractive summarization systems.
• To overcome the poor supervision signal in reference summaries, which reflect a trade-off between relevance and saliency.
• To improve query relevance measurement beyond surface-level features like TF-IDF, which fail to capture semantic intent.
• To develop a data-driven, end-to-end model that learns meaningful sentence and document representations automatically.
• To simulate human attentive reading by focusing on sentences that best answer the query.

Proposed method

• The model uses a neural network to learn distributed sentence embeddings and document cluster embeddings via weighted-sum pooling.
• It applies an attention mechanism that computes query-dependent weights for each sentence, emphasizing those most relevant to the query.
• The document representation is formed by a weighted sum of sentence embeddings, where weights are derived from the attention mechanism's relevance scores.
• The attention mechanism dynamically focuses on sentences that align semantically with both the query and the document content.
• The model is trained end-to-end using reference summaries as supervision, with no hand-crafted features required.
• Sentence saliency is implicitly captured through the attention mechanism, as highly relevant sentences contribute more to the document representation.

Experimental results

Research questions

• RQ1 Can a joint model improve both query relevance and sentence saliency ranking compared to isolated training?
• RQ2 Can neural attention mechanisms effectively simulate human attentive reading in query-focused summarization?
• RQ3 Does a data-driven, end-to-end model without hand-crafted features outperform systems relying on engineered features?
• RQ4 Are the sentences with high attention scores semantically aligned with the query intent, even when key terms are absent?
• RQ5 Can the learned document representations support future extension to abstractive summarization?

Key findings

• AttSum achieves competitive performance on the DUC 2005–2007 query-focused summarization benchmarks without using any hand-crafted features.
• The model successfully identifies sentences that meet the query need, even when key query terms are absent from the content, indicating semantic understanding beyond surface-level matching.
• Sentences with high attention scores are consistently relevant to the query intent, as shown in qualitative analysis, such as correctly identifying benefits of drug legalization despite low TF-IDF overlap.
• The attention mechanism effectively biases the document representation toward sentences that are both relevant to the query and representative of the document cluster.
• The joint learning of relevance and saliency through attention leads to better performance than isolated models, due to more effective supervision from reference summaries.
• The system's ability to generate meaningful semantic representations suggests potential for extension to abstractive summarization with a neural language model.