[Paper Review] Uncertainty in Neural Network Word Embedding: Exploration of Threshold for Similarity

Navid Rekabsaz, Mihai Lupu+1 more|arXiv (Cornell University)|Jun 20, 2016

Topic Modeling26 references3 citations

TL;DR

This paper proposes a data-driven, uncertainty-aware threshold for identifying semantically related words in neural network word embeddings, using a continuous neighborhood representation to estimate a stable, generalizable cutoff. Evaluated on four IR collections, the method achieves performance statistically indistinguishable from the optimal threshold across all dimensions (100–400), significantly outperforming baseline k-NN approaches.

ABSTRACT

Word embedding, specially with its recent developments, promises a quantification of the similarity between terms. However, it is not clear to which extent this similarity value can be genuinely meaningful and useful for subsequent tasks. We explore how the similarity score obtained from the models is really indicative of term relatedness. We first observe and quantify the uncertainty factor of the word embedding models regarding to the similarity value. Based on this factor, we introduce a general threshold on various dimensions which effectively filters the highly related terms. Our evaluation on four information retrieval collections supports the effectiveness of our approach as the results of the introduced threshold are significantly better than the baseline while being equal to or statistically indistinguishable from the optimal results.

Motivation & Objective

• To address the lack of principled methods for identifying truly semantically related terms in word embeddings, beyond arbitrary k-NN or fixed thresholds.
• To quantify the uncertainty in similarity scores produced by neural network word embeddings due to stochastic training and initialization.
• To develop a general, dimension-agnostic threshold that separates semantically related terms from unrelated ones, independent of individual terms.
• To evaluate the threshold’s effectiveness in query expansion for document retrieval, comparing it to k-NN and optimal threshold baselines.
• To demonstrate that the proposed threshold yields results statistically indistinguishable from the optimal threshold across multiple IR test collections.

Proposed method

• The method models the uncertainty in word embedding similarity by training multiple instances of the same model on the same data with different random seeds, capturing variance in similarity scores.
• It constructs a continuous neighborhood representation for each word by aggregating similarity scores across multiple model runs, forming a distribution of neighbor scores.
• A threshold is derived from the central tendency and confidence interval of this uncertainty-aware neighborhood distribution, ensuring robustness to model variance.
• The threshold is applied to filter only highly reliable, semantically related terms during query expansion in document retrieval.
• The approach is evaluated using standard IR metrics (MAP and NDCG) on four benchmark collections, comparing performance against k-NN and optimal threshold baselines.
• The method uses skip-gram with negative sampling (SGNS) in the Word2Vec framework, with embeddings trained on 100–400 dimensions.

Experimental results

Research questions

• RQ1 Can uncertainty in word embedding similarity scores be quantified and leveraged to define a stable threshold for identifying semantically related terms?
• RQ2 Does a general, dimension-agnostic threshold derived from uncertainty modeling outperform k-NN and fixed-threshold baselines in information retrieval tasks?
• RQ3 How does the performance of the proposed threshold compare to the optimal threshold found via exhaustive search across different embedding dimensions?
• RQ4 To what extent does the distribution of related terms per word (e.g., number of neighbors) match known linguistic resources like WordNet?

Key findings

• The proposed uncertainty-aware threshold achieved performance statistically indistinguishable from the optimal threshold across all tested embedding dimensions (100, 200, 300, 400) on four IR collections.
• The method significantly outperformed the k-NN baseline, which deteriorated rapidly with increasing k, performing only slightly better than baseline for k=1 and k=2.
• For 300-dimensional embeddings, the mean number of similar terms per query was 1.5 with a standard deviation of 3.0, closely matching WordNet’s mean of 1.6 and standard deviation of 3.1.
• The threshold was optimal for 100, 200, and 300 dimensions, and statistically indistinguishable from optimal at 400 dimensions, demonstrating robustness across dimensions.
• The results show a clear performance peak around the proposed threshold, with performance degrading when the threshold is set too low (noise) or too high (over-filtering).
• The uncertainty-based neighborhood representation effectively captures the intrinsic variability in word embedding similarity, enabling a principled, generalizable thresholding strategy.