[Paper Review] Variations of the Similarity Function of TextRank for Automated Summarization

Federico Barrios, Federico López+2 more|arXiv (Cornell University)|Feb 11, 2016

Topic Modeling23 references114 citations

TL;DR

The paper proposes alternative sentence-similarity measures for TextRank and shows that BM25/BM25+ based edge weighting improves ROUGE scores on the DUC2002 dataset, with BM25 achieving the best reported gain.

ABSTRACT

This article presents new alternatives to the similarity function for the TextRank algorithm for automatic summarization of texts. We describe the generalities of the algorithm and the different functions we propose. Some of these variants achieve a significative improvement using the same metrics and dataset as the original publication.

Motivation & Objective

• Enhance TextRank for extractive summarization by modifying how sentence similarity is computed.
• Evaluate alternative similarity functions on a standard benchmark (DUC 2002).
• Identify variants that improve ROUGE metrics without extra supervision or training.
• Assess computational efficiency of the proposed variants.

Proposed method

• Represent documents as sentence graphs where edge weights are derived from sentence similarity.
• Propose and implement multiple similarity variants: Longest Common Substring, Cosine TF-IDF, BM25, and BM25+.
• Apply a corrective IDF formulation to BM25 variants to stabilize low-frequency terms.
• Evaluate using ROUGE-1, ROUGE-2, and ROUGE-SU4 on the DUC 2002 corpus.
• Compare against the original TextRank baseline and report percentage improvements.
• Provide reference implementation and contributions to Gensim.

Experimental results

Research questions

• RQ1 Do alternative similarity measures for edge weighting in TextRank improve extractive summarization quality?
• RQ2 Which similarity variant yields the best ROUGE gains on the DUC 2002 dataset?
• RQ3 How do BM25-based variants compare to traditional TF-IDF or overlap-based measures in TextRank?
• RQ4 What is the trade-off between quality gains and computation time for the proposed variants?

Key findings

• BM25 and BM25+ with a specific corrective formula yield the highest ROUGE gains, improving over the original TextRank by 2.92%.
• Cosine TF-IDF also yields a notable improvement of 2.54% over the original TextRank.
• Longest Common Substring provides a modest 1.40% improvement over TextRank.
• BM25 and BM25+ variants generally outperform several other variants and baselines on ROUGE metrics.
• The best-performing setup (BM25 with ε=0.25) also processes the DUC 567-document corpus faster than the original TextRank (84% of the original time).
• The authors provide a reference Python implementation and contributed BM25-TextRank to Gensim.