[Paper Review] WordRank: Learning Word Embeddings via Robust Ranking

Shihao Ji, Hyokun Yun+3 more|arXiv (Cornell University)|Jun 9, 2015

Topic Modeling32 references4 citations

TL;DR

WordRank proposes a novel word embedding framework that formulates word representation learning as a robust ranking problem, using DCG-like ranking losses to naturally incorporate attention and noise robustness. It achieves state-of-the-art performance on word similarity and analogy tasks, especially under low-data regimes—matching models trained on 7.2B tokens with just 17M tokens.

ABSTRACT

Embedding words in a vector space has gained a lot of attention in recent years. While state-of-the-art methods provide efficient computation of word similarities via a low-dimensional matrix embedding, their motivation is often left unclear. In this paper, we argue that word embedding can be naturally viewed as a ranking problem due to the ranking nature of the evaluation metrics. Then, based on this insight, we propose a novel framework WordRank that efficiently estimates word representations via robust ranking, in which the attention mechanism and robustness to noise are readily achieved via the DCG-like ranking losses. The performance of WordRank is measured in word similarity and word analogy benchmarks, and the results are compared to the state-of-the-art word embedding techniques. Our algorithm is very competitive to the state-of-the- arts on large corpora, while outperforms them by a significant margin when the training set is limited (i.e., sparse and noisy). With 17 million tokens, WordRank performs almost as well as existing methods using 7.2 billion tokens on a popular word similarity benchmark. Our multi-node distributed implementation of WordRank is publicly available for general usage.

Motivation & Objective

• To address the limitations of existing word embedding methods that optimize co-occurrence counts directly, which lack inherent noise robustness and attention mechanisms.
• To reframe word embedding as a ranking problem, aligning more naturally with evaluation metrics like word similarity and analogy.
• To develop a method that maintains high performance even when training data are limited or noisy, where standard methods degrade significantly.
• To enable efficient, scalable training via a multi-node distributed implementation, publicly released for community use.

Proposed method

• WordRank models word-context relationships by ranking context words based on their co-occurrence relevance, where the goal is to rank semantically similar contexts higher.
• It uses a differentiable, concave ranking loss function ρ(rank(w,c)) that is monotonically increasing, enabling optimization of relative ordering rather than absolute values.
• The ranking loss inherently provides an attention mechanism by emphasizing top-ranked contexts and down-weighting less relevant ones.
• It achieves robustness to noise by focusing on relative ranking order, reducing sensitivity to spurious co-occurrences in small or noisy corpora.
• The method is implemented using a distributed framework, allowing efficient training on large-scale corpora with minimal hyperparameter tuning.
• WordRank uses the same co-occurrence matrix as baseline methods (e.g., GloVe and word2vec), ensuring fair comparison while optimizing a ranking objective instead of log-likelihood or PMI.

Experimental results

Research questions

• RQ1 Can word representation learning be effectively reframed as a ranking problem to better align with word similarity and analogy evaluation metrics?
• RQ2 Does a ranking-based objective improve performance on word similarity and analogy tasks, especially when training data are sparse or noisy?
• RQ3 Can a ranking loss naturally induce attention and robustness to noise without requiring explicit architectural modifications?
• RQ4 How does WordRank compare to state-of-the-art methods like word2vec and GloVe in terms of performance across varying corpus sizes?

Key findings

• With only 17 million tokens, WordRank achieves performance on the word similarity benchmark that is nearly equivalent to models trained on 7.2 billion tokens.
• On the word similarity task, WordRank consistently outperforms both word2vec and GloVe across all corpus sizes, especially in low-data regimes.
• On the word analogy task, WordRank performs competitively with word2vec and GloVe when corpus size is large, but maintains an advantage in smaller settings.
• WordRank outperforms both baselines on 5 out of 6 word similarity datasets and on 1 out of 2 subtasks of the Google analogy dataset.
• t-SNE visualization of WordRank embeddings shows meaningful clustering of semantic (e.g., feline, kitten) and syntactic (e.g., leashed, leashes) neighbors of 'cat'.
• The multi-node distributed implementation of WordRank is publicly available, enabling scalable deployment and reproducibility.