[Paper Review] Simple Unsupervised Keyphrase Extraction using Sentence Embeddings

Kamil Bennani-Smires, Claudiu Musat+3 more|arXiv (Cornell University)|Jan 13, 2018

Advanced Text Analysis Techniques4 citations

TL;DR

This paper proposes EmbedRank, an unsupervised keyphrase extraction method that uses sentence embeddings to select informative and diverse keyphrases from a single document without requiring a corpus. It outperforms state-of-the-art graph-based methods on standard datasets and demonstrates through a user study that diversity-enhanced keyphrases are preferred by humans, despite no improvement in F-score.

ABSTRACT

Keyphrase extraction is the task of automatically selecting a small set of phrases that best describe a given free text document. Supervised keyphrase extraction requires large amounts of labeled training data and generalizes very poorly outside the domain of the training data. At the same time, unsupervised systems have poor accuracy, and often do not generalize well, as they require the input document to belong to a larger corpus also given as input. Addressing these drawbacks, in this paper, we tackle keyphrase extraction from single documents with EmbedRank: a novel unsupervised method, that leverages sentence embeddings. EmbedRank achieves higher F-scores than graph-based state of the art systems on standard datasets and is suitable for real-time processing of large amounts of Web data. With EmbedRank, we also explicitly increase coverage and diversity among the selected keyphrases by introducing an embedding-based maximal marginal relevance (MMR) for new phrases. A user study including over 200 votes showed that, although reducing the phrases' semantic overlap leads to no gains in F-score, our high diversity selection is preferred by humans.

Motivation & Objective

• Address the limitations of supervised keyphrase extraction, which require large labeled datasets and generalize poorly across domains.
• Overcome the drawbacks of existing unsupervised methods that either depend on a large corpus or suffer from redundancy and low diversity.
• Develop a fast, scalable, and corpus-independent method for real-time keyphrase extraction suitable for web-scale data.
• Improve keyphrase diversity and user preference without sacrificing F-score, challenging the adequacy of F-score as the sole evaluation metric.

Proposed method

• Leverage pre-trained sentence embeddings (e.g., Sent2Vec, Doc2Vec) to represent both the full document and candidate phrases in a shared vector space.
• Compute informativeness by measuring the semantic distance between a candidate phrase’s embedding and the document’s embedding.
• Apply an embedding-based maximal marginal relevance (MMR) strategy to balance informativeness and diversity during keyphrase selection.
• Use a parameterized MMR formulation with λ=0.5 to control redundancy, favoring phrases that are both relevant and semantically distinct from already selected ones.
• Aggregate word scores into phrase scores using a weighted sum of word embeddings, avoiding over-generation from individual word centrality.
• Implement the method as a simple, scalable pipeline that operates solely on the input document, without requiring external corpora.

Experimental results

Research questions

• RQ1 Can an unsupervised keyphrase extraction method achieve higher F-scores than existing graph-based state-of-the-art methods without relying on a large training corpus?
• RQ2 Does explicitly controlling redundancy through embedding-based MMR improve human preference for extracted keyphrases, even if F-score remains unchanged?
• RQ3 To what extent does the use of phrase-level embeddings instead of word-level embeddings improve keyphrase quality and diversity?
• RQ4 Can a corpus-free method based on sentence embeddings be both fast and effective for real-time processing of web-scale text data?
• RQ5 Is F-score a sufficient or reliable evaluation metric for keyphrase quality when human preference favors diverse, non-redundant outputs?

Key findings

• EmbedRank achieves higher F-scores than state-of-the-art graph-based methods (e.g., TextRank, SingleRank, WordAttractionRank) on two out of three standard keyphrase extraction datasets.
• The user study with 214 votes showed that 70% of participants preferred keyphrases extracted with EmbedRank++ (λ=0.5) over those from EmbedRank (λ=1), indicating strong human preference for diversity.
• Despite the user preference for diverse keyphrases, EmbedRank++ resulted in a measurable decrease in F-score, suggesting a disconnect between automatic evaluation and human judgment.
• The method is scalable and efficient, with Sent2Vec enabling fast inference, making it suitable for real-time processing of large volumes of web and social media data.
• EmbedRank++ reduces redundancy by selecting semantically distinct phrases, even when those phrases are not in the gold-standard keyphrases, highlighting the risk of F-score penalizing semantically equivalent but non-identical keyphrases.
• The study raises critical questions about the reliability of F-score as the primary evaluation metric for keyphrase extraction, advocating for new evaluation methodologies that incorporate human preferences.