[Paper Review] Keywords lie far from the mean of all words in local vector space

Eirini Papagiannopoulou, Grigorios Tsoumakas+1 more|arXiv (Cornell University)|Aug 21, 2020

Advanced Text Analysis Techniques47 references4 citations

TL;DR

This paper proposes an unsupervised keyword extraction method that models document word distributions using local word vector representations, ranking candidates based on their distance from the distribution's center (mean vector) and first-occurrence position. It shows that keywords lie far from the mean of non-keyword terms, outperforming state-of-the-art unsupervised methods by leveraging Euclidean and Mahalanobis distances over cosine similarity.

ABSTRACT

Keyword extraction is an important document process that aims at finding a small set of terms that concisely describe a document's topics. The most popular state-of-the-art unsupervised approaches belong to the family of the graph-based methods that build a graph-of-words and use various centrality measures to score the nodes (candidate keywords). In this work, we follow a different path to detect the keywords from a text document by modeling the main distribution of the document's words using local word vector representations. Then, we rank the candidates based on their position in the text and the distance between the corresponding local vectors and the main distribution's center. We confirm the high performance of our approach compared to strong baselines and state-of-the-art unsupervised keyword extraction methods, through an extended experimental study, investigating the properties of the local representations.

Motivation & Objective

• To develop an unsupervised keyword extraction method that avoids reliance on external tools or pre-trained embeddings.
• To investigate whether local word vector representations can effectively model the semantic center of a document’s non-keyword terms.
• To evaluate the effectiveness of distance metrics like Euclidean and Mahalanobis over cosine similarity in distinguishing keywords from non-keywords.
• To demonstrate that keywords are statistically distant from the main distribution of words in local vector space.
• To explore the impact of robust statistics and positional features on keyword ranking performance.

Proposed method

• Construct local word vector representations using a sliding window around each word, capturing co-occurrence patterns.
• Compute the mean vector (distribution center) of all words in the document, representing the main semantic bulk of non-keywords.
• Score each candidate keyword using a composite score based on its Euclidean or Mahalanobis distance from the mean vector and its first-occurrence position in the text.
• Use Mahalanobis distance to account for covariance structure in word vector distributions, improving discrimination of outliers (keywords).
• Apply robust estimators (e.g., MCD) to compute the center and covariance matrix, reducing sensitivity to outliers in the estimation phase.
• Rank keywords by their composite score, with higher values indicating greater relevance.

Experimental results

Research questions

• RQ1 Do keywords in a document tend to lie far from the mean of all word vectors in local vector space?
• RQ2 Can simpler local vector representations (based on co-occurrence within a window) match the performance of pre-trained embeddings like GloVe in keyword extraction?
• RQ3 Does incorporating vector magnitude or covariance (via Mahalanobis distance) improve keyword detection over cosine-based methods?
• RQ4 How does the position of first occurrence interact with vector distance to improve keyword ranking?
• RQ5 Can robust statistics enhance the estimation of the distribution center and improve overall keyword extraction performance?

Key findings

• Keywords are consistently located far from the mean vector of all words in local vector space, validating the core intuition of the method.
• The proposed method, LV b, retrieves 6 out of 6 keywords in the top 15 results on Article I, outperforming both LV and FWN baselines.
• On Article II, LV b fails to retrieve 50% of keywords in the top 15, but the full LV method successfully retrieves the keyword 'dynamic' due to early occurrence and high distance score.
• Local vectors based solely on co-occurrence within a window encode sufficient statistical information to match or exceed performance of pre-trained GloVe embeddings.
• Using Mahalanobis distance significantly improves performance over cosine similarity, especially in capturing the distinct behavior of keyword vectors.
• Robust estimation of the center and covariance matrix enhances stability and performance, particularly in the presence of noisy or outlier-rich distributions.