[Paper Review] Outperforming Word2Vec on Analogy Tasks with Random Projections

Abram Demski, Volkan Ustun+2 more|arXiv (Cornell University)|Dec 20, 2014

Topic Modeling11 references3 citations

TL;DR

This paper introduces a simple, non-neural vector representation method based on random projections that outperforms Word2Vec on analogy tasks. By summing random vectors and their pointwise products, the approach achieves state-of-the-art performance without training, matrix decompositions, or convolutions, effectively leveraging random projection for dimensionality reduction.

ABSTRACT

We present a distributed vector representation based on a simplification of the BEAGLE system, designed in the context of the Sigma cognitive architecture. Our method does not require gradient-based training of neural networks, matrix decompositions as with LSA, or convolutions as with BEAGLE. All that is involved is a sum of random vectors and their pointwise products. Despite the simplicity of this technique, it gives state-of-the-art results on analogy problems, in most cases better than Word2Vec. To explain this success, we interpret it as a dimension reduction via random projection.

Motivation & Objective

• To develop a simple, non-neural method for distributed vector representations that avoids complex training procedures.
• To improve performance on analogy tasks compared to existing methods like Word2Vec and LSA.
• To investigate whether random projections can serve as an effective dimensionality reduction technique for word representations.
• To eliminate the need for gradient-based optimization, matrix decompositions, or convolutional operations in word embedding learning.

Proposed method

• The method constructs word vectors as the sum of random vectors and their pointwise products.
• Each word is represented by a fixed-size vector formed through random projection without learning.
• The technique relies on the statistical properties of random projections to preserve semantic structure.
• No backpropagation, matrix factorization, or convolutional layers are used in the process.
• The final representation is derived purely from random initialization and element-wise operations.
• The approach is inspired by the BEAGLE system but simplified to remove complex components.

Experimental results

Research questions

• RQ1 Can a simple, non-neural method based on random projections outperform neural network-based models like Word2Vec on analogy tasks?
• RQ2 Does random projection effectively serve as a dimensionality reduction technique for word representations?
• RQ3 What is the performance of a method that avoids gradient-based training and matrix decompositions on semantic analogy tasks?
• RQ4 How does the sum of random vectors and their pointwise products compare to established word embedding techniques?

Key findings

• The proposed method achieves state-of-the-art results on analogy tasks, surpassing Word2Vec in most cases.
• The method does not require any training, matrix decomposition, or convolutional operations.
• Performance is attributed to the effectiveness of random projection in preserving semantic relationships.
• The simplicity of the method—based solely on random vectors and their products—yields strong empirical results.