[Paper Review] Combining Word Embeddings and N-grams for Unsupervised Document Summarization

Zhuolin Jiang, Manaj Srivastava+3 more|arXiv (Cornell University)|Apr 25, 2020

Topic Modeling28 references4 citations

TL;DR

This paper proposes an unsupervised, extractive document summarization method that enhances sentence similarity via a fusion of pre-trained word embeddings (word2vec, BERT) and n-gram features, improving graph-based submodular summarization. It further introduces a Transformer-based sentence compression model that boosts performance on DUC04, achieving state-of-the-art results without supervision and comparable performance to supervised models on CNN/DM and NYT datasets.

ABSTRACT

Graph-based extractive document summarization relies on the quality of the sentence similarity graph. Bag-of-words or tf-idf based sentence similarity uses exact word matching, but fails to measure the semantic similarity between individual words or to consider the semantic structure of sentences. In order to improve the similarity measure between sentences, we employ off-the-shelf deep embedding features and tf-idf features, and introduce a new text similarity metric. An improved sentence similarity graph is built and used in a submodular objective function for extractive summarization, which consists of a weighted coverage term and a diversity term. A Transformer based compression model is developed for sentence compression to aid in document summarization. Our summarization approach is extractive and unsupervised. Experiments demonstrate that our approach can outperform the tf-idf based approach and achieve state-of-the-art performance on the DUC04 dataset, and comparable performance to the fully supervised learning methods on the CNN/DM and NYT datasets.

Motivation & Objective

• To improve sentence similarity measurement in graph-based extractive summarization by replacing traditional tf-idf with deep semantic features.
• To investigate the effectiveness of combining pre-trained word embeddings (word2vec, BERT) with n-gram features for better semantic representation in unsupervised summarization.
• To evaluate the impact of sentence compression using a Transformer model on extractive summarization performance.
• To achieve state-of-the-art or competitive performance on benchmark datasets without requiring labeled summary pairs.

Proposed method

• Constructs a sentence similarity graph using a novel metric that fuses tf-idf and pre-trained word embeddings (word2vec, BERT) to improve semantic similarity estimation.
• Applies a submodular objective function combining weighted coverage and diversity terms to select representative and diverse sentences under a budget constraint.
• Uses a greedy algorithm to maximize the submodular objective, with sentence cost normalized via local neighborhood statistics.
• Trains a Transformer-based model on Gigaword and Google sentence compression datasets to compress input sentences before summarization.
• Employs a late fusion strategy to combine word embeddings and n-gram features in the sentence similarity computation.
• Uses ROUGE scores (R-1, R-2, R-L) for evaluation on DUC04, CNN/DM, and NYT50 datasets with a three-sentence summary budget.

Experimental results

Research questions

• RQ1 Can combining off-the-shelf word embeddings with n-gram features improve sentence similarity estimation in unsupervised summarization?
• RQ2 Does the fusion of deep semantic features (word2vec, BERT) with sparse tf-idf features lead to better summarization performance than using either alone?
• RQ3 To what extent does applying a Transformer-based sentence compression model enhance extractive summarization performance?
• RQ4 Can an unsupervised summarization system achieve performance comparable to supervised state-of-the-art methods on CNN/DM and NYT datasets?

Key findings

• The proposed GraphFusion method, which combines word2vec and tf-idf features, achieves a 39.0 R-1, 16.8 R-2, and 32.0 R-L F-score on the CNN/DM dataset, outperforming tf-idf and other embedding-only baselines.
• On the DUC04 dataset, the method achieves a 1-point absolute improvement in all three ROUGE metrics (R-1, R-2, R-L) over current baselines by using the Google sentence compression dataset in addition to Gigaword.
• The LateFusion approach, which combines word2vec and n-gram features, achieves the best performance on CNN/DM with 39.2 R-1, 17.1 R-2, and 32.2 R-L F-scores.
• The Transformer-based sentence compression model improves summarization performance on DUC04, with absolute gains of nearly 2 points on ROUGE-1, ROUGE-2, and ROUGE-L over baseline compression models.
• Despite being unsupervised and not using training data, the method achieves performance comparable to supervised models like Pointer and Refresh on CNN/DM and NYT datasets.
• The use of sentence compression did not improve results on CNN/DM, likely due to information loss under a fixed three-sentence budget, but significantly improved performance on DUC04.