[Paper Review] Unsupervised Extractive Summarization using Pointwise Mutual Information

Vishakh Padmakumar, He He|arXiv (Cornell University)|Feb 11, 2021

Topic Modeling26 references5 citations

TL;DR

This paper proposes an unsupervised extractive summarization method that uses pointwise mutual information (PMI) between sentences to measure relevance to the document and redundancy within the summary. By leveraging a fine-tuned pre-trained language model to estimate PMI, the method selects sentences greedily to maximize relevance and minimize redundancy, outperforming similarity-based baselines across diverse domains including news, medical articles, and personal anecdotes.

ABSTRACT

Unsupervised approaches to extractive summarization usually rely on a notion of sentence importance defined by the semantic similarity between a sentence and the document. We propose new metrics of relevance and redundancy using pointwise mutual information (PMI) between sentences, which can be easily computed by a pre-trained language model. Intuitively, a relevant sentence allows readers to infer the document content (high PMI with the document), and a redundant sentence can be inferred from the summary (high PMI with the summary). We then develop a greedy sentence selection algorithm to maximize relevance and minimize redundancy of extracted sentences. We show that our method outperforms similarity-based methods on datasets in a range of domains including news, medical journal articles, and personal anecdotes.

Motivation & Objective

• To address the lack of labeled data in non-news domains by developing an unsupervised extractive summarization method that does not rely on sentence similarity.
• To formalize sentence importance in summarization using PMI as a measure of relevance (sentence-document dependence) and redundancy (sentence-sentence dependence).
• To design a greedy sentence selection algorithm that maximizes relevance and minimizes redundancy using PMI scores estimated via a fine-tuned language model.
• To demonstrate generalization across domains, including news, medical articles, and personal narratives, where similarity-based methods often fail.
• To reduce reliance on position bias, which plagues supervised models and some extractive baselines like PacSum.

Proposed method

• Relevance is defined as the PMI between a summary sentence and the document, computed as log(P(d|s)/P(d)) using a language model to estimate conditional and marginal probabilities.
• Redundancy is measured as the PMI between pairs of summary sentences, with the conditional probability P(s_j|s_i) used to assess whether a later sentence can be inferred from an earlier one.
• The method uses a fine-tuned pre-trained language model (e.g., BERT) to estimate PMI scores efficiently and contextually, avoiding reliance on lexical or embedding similarity.
• A greedy sentence selection algorithm sequentially adds sentences that maximize the gain in relevance while minimizing redundancy, based on cumulative PMI scores.
• The approach is position-agnostic, avoiding the lead bias that plagues models like PacSum, which perform poorly on non-news datasets.
• The method is evaluated using ROUGE scores across CNN/DM, Reddit-TIFU, and medical article datasets, with ablation studies isolating the contributions of relevance and redundancy.

Experimental results

Research questions

• RQ1 Can PMI-based metrics for relevance and redundancy outperform traditional sentence similarity-based features in unsupervised extractive summarization?
• RQ2 To what extent does combining relevance and redundancy in the selection criterion improve summarization performance across diverse domains?
• RQ3 How does the proposed method compare to position-based baselines like PacSum and lead-k, especially in the absence of lead bias?
• RQ4 Does the use of a pre-trained language model to estimate PMI lead to better content selection than lexical or embedding-based similarity?
• RQ5 What is the relative contribution of relevance versus redundancy in the final summarization performance?

Key findings

• The proposed method outperforms similarity-based baselines on CNN/DM, Reddit-TIFU, and medical article datasets, achieving a ROUGE-1 score of 36.68 on CNN/DM and 18.93 on Reddit-TIFU.
• Ablation studies show that combining relevance and redundancy yields the best performance, with relevance alone achieving 35.17 ROUGE-1 on CNN/DM and redundancy alone performing poorly at 23.85.
• The method is position-agnostic and avoids the lead bias that advantages PacSum on CNN/DM, where 82.3% of PacSum’s selected sentences were in the first three positions, compared to only 21.4% for the proposed method.
• The performance gap between PacSum and other extractive methods on CNN/DM is largely due to position bias, as PacSum’s performance degrades significantly on XSum, which lacks such bias.
• The method generalizes well across domains, including non-news domains like personal anecdotes and medical articles, where similarity-based methods often fail.
• The use of PMI as a measure of semantic dependence leads to more meaningful content selection than lexical or embedding similarity, as shown by consistent gains in ROUGE scores.