[Paper Review] Toward Incorporation of Relevant Documents in word2vec

Navid Rekabsaz, Bhaskar Mitra+2 more|arXiv (Cornell University)|Jul 20, 2017

Topic Modeling26 references6 citations

TL;DR

This paper proposes a novel, interpretable explicit word representation derived from the Skip-Gram model that captures genuine co-occurrence probabilities, enabling effective integration of local query-relevant documents into word embeddings. The method outperforms state-of-the-art explicit representations in retrieving highly similar terms and introduces five functions to adapt global word vectors using local document content, with preliminary results showing improved relevance alignment.

ABSTRACT

Recent advances in neural word embedding provide significant benefit to various information retrieval tasks. However as shown by recent studies, adapting the embedding models for the needs of IR tasks can bring considerable further improvements. The embedding models in general define the term relatedness by exploiting the terms' co-occurrences in short-window contexts. An alternative (and well-studied) approach in IR for related terms to a query is using local information i.e. a set of top-retrieved documents. In view of these two methods of term relatedness, in this work, we report our study on incorporating the local information of the query in the word embeddings. One main challenge in this direction is that the dense vectors of word embeddings and their estimation of term-to-term relatedness remain difficult to interpret and hard to analyze. As an alternative, explicit word representations propose vectors whose dimensions are easily interpretable, and recent methods show competitive performance to the dense vectors. We introduce a neural-based explicit representation, rooted in the conceptual ideas of the word2vec Skip-Gram model. The method provides interpretable explicit vectors while keeping the effectiveness of the Skip-Gram model. The evaluation of various explicit representations on word association collections shows that the newly proposed method out- performs the state-of-the-art explicit representations when tasked with ranking highly similar terms. Based on the introduced ex- plicit representation, we discuss our approaches on integrating local documents in globally-trained embedding models and discuss the preliminary results.

Motivation & Objective

• To address the lack of interpretability in dense word embeddings used in IR, especially when topic shifting occurs due to global co-occurrence patterns.
• To develop an explicit word representation that retains the effectiveness of Skip-Gram while enabling clear interpretation of vector dimensions.
• To explore efficient methods for incorporating local document information (e.g., top-k retrieved documents) into pre-trained word embeddings for improved query-specific term relatedness.
• To evaluate whether integrating local relevance signals via interpretable vectors improves performance on word association and similarity tasks.

Proposed method

• Proposes a new explicit word representation where each dimension corresponds to a context term, and vector values represent the probability of genuine co-occurrence between the word and context term, derived from the Skip-Gram model.
• Uses a logistic function to reweight global co-occurrence values $v(w,c)$ to $\hat{v}(w,c)$ based on local document relevance, with learned parameters $a$ and $b$.
• Defines five functions $f_1$ to $f_5$ to incorporate local document information: (1) binary presence of context terms, (2) relative term frequency in local vs. global corpora, (3) local-to-global co-occurrence ratio, (4) relevance model using document language models, and (5) joint probability of word and context under document models.
• Employs Dirichlet smoothing with $\mu=1500$ to estimate language models for documents in the top-k retrieved set.
• Treats the vector representation as interpretable by design, with dimensions corresponding to specific context terms, enabling causal analysis and topic injection.
• Evaluates the method on word association benchmarks, focusing on top-k similar term retrieval performance.

Experimental results

Research questions

• RQ1 Can an explicit word representation be designed to preserve the effectiveness of Skip-Gram while enabling full interpretability of vector dimensions?
• RQ2 How can local document information (e.g., top-k retrieved documents) be effectively and efficiently incorporated into pre-trained word embeddings to improve query-specific term relatedness?
• RQ3 Which function for integrating local relevance signals (e.g., term frequency, co-occurrence, language models) yields the most effective adaptation of word vectors for IR tasks?
• RQ4 Does the proposed method outperform existing explicit representations in retrieving highly similar terms to a query word?

Key findings

• The proposed explicit representation achieves performance comparable to state-of-the-art explicit methods on word association benchmarks.
• The method outperforms existing explicit representations in the task of retrieving the top-similar terms to a given word, indicating improved sensitivity to semantic similarity.
• The $f_1$ method, which only considers presence of context terms in local documents, showed minimal improvement, suggesting that term frequency and weighting are crucial for relevance filtering.
• The $f_2$ and $f_3$ functions, which use relative frequency and co-occurrence ratios, respectively, showed stronger potential for improving vector adaptation.
• The $f_4$ and $f_5$ functions, based on relevance models and joint probabilities under document language models, demonstrated the most promising results by incorporating both query and document context.
• Preliminary results suggest that integrating local document content via interpretable vectors can effectively reduce topic drift and improve relevance alignment in IR tasks.