[Paper Review] Word2Vec is a special case of Kernel Correspondence Analysis and Kernels for Natural Language Processing

Hirotaka Niitsuma, Minho Lee|arXiv (Cornell University)|May 17, 2016

Natural Language Processing Techniques4 citations

TL;DR

This paper establishes Word2Vec as a special case of Kernel Correspondence Analysis (KCA), introducing a memory-efficient, delayed sparse randomized SVD (DSSVD) algorithm to enable CA on large-scale NLP tasks. It proposes a novel 'tail-cut kernel' that outperforms existing word-vector methods by filtering noisy co-occurrences, achieving state-of-the-art performance on word similarity benchmarks while maintaining deterministic results and reduced hyperparameter sensitivity.

ABSTRACT

We show that correspondence analysis (CA) is equivalent to defining a Gini index with appropriately scaled one-hot encoding. Using this relation, we introduce a nonlinear kernel extension to CA. This extended CA gives a known analysis for natural language via specialized kernels that use an appropriate contingency table. We propose a semi-supervised CA, which is a special case of the kernel extension to CA. Because CA requires excessive memory if applied to numerous categories, CA has not been used for natural language processing. We address this problem by introducing delayed evaluation to randomized singular value decomposition. The memory-efficient CA is then applied to a word-vector representation task. We propose a tail-cut kernel, which is an extension to the skip-gram within the kernel extension to CA. Our tail-cut kernel outperforms existing word-vector representation methods.

Motivation & Objective

• To address the memory and computational infeasibility of applying Correspondence Analysis (CA) to large-scale NLP tasks with thousands of categories.
• To establish a theoretical link between Word2Vec and CA, showing Word2Vec as a special case of Kernel Correspondence Analysis (KCA).
• To develop a memory-efficient, deterministic alternative to neural network-based word vector models like Word2Vec, which suffer from random initialization and hyperparameter sensitivity.
• To propose a novel 'tail-cut kernel' that improves word vector quality by filtering spurious co-occurrences in the context window.
• To demonstrate that kernelized CA can outperform state-of-the-art word representation methods on standard similarity benchmarks.

Proposed method

• Proposes a delayed sparse randomized SVD (DSSVD) algorithm that avoids dense matrix expansion, drastically reducing memory usage and computation time for large contingency tables.
• Reframes CA as PCA on a contingency table, and shows that CA is equivalent to a Gini index-based variance measure on one-hot encoded categorical data.
• Introduces a nonlinear kernel extension to CA (KCA) using specialized kernels derived from co-occurrence statistics, enabling nonlinear feature extraction.
• Develops the 'tail-cut kernel' by applying a thresholding rule to co-occurrence counts, retaining only statistically significant word pairs to reduce noise.
• Applies KCA with the tail-cut kernel to word co-occurrence matrices, producing deterministic, high-quality word vectors without random initialization.
• Uses SVD-based LCA and KCA on different co-occurrence matrices: N^skip (skip-gram like), N^flat (flat window), and N^cut (tail-cut kernel), comparing performance.

Experimental results

Research questions

• RQ1 Can Correspondence Analysis (CA) be made scalable to large-scale NLP tasks with tens of thousands of word categories?
• RQ2 Is Word2Vec mathematically equivalent to a specific instance of Kernel Correspondence Analysis (KCA)?
• RQ3 Can a kernelized extension of CA outperform neural network-based word vector models like Word2Vec and GloVe?
• RQ4 Does the proposed tail-cut kernel reduce sensitivity to window size while improving performance on word similarity tasks?
• RQ5 Can semi-supervised CA (SCA) leverage labeled data to further improve word vector quality in an otherwise unsupervised setting?

Key findings

• The DSSVD algorithm reduced memory usage to 10% of standard SVD and 20% of randomized SVD, while accelerating computation by 100x compared to standard SVD.
• The tail-cut kernel achieved the best or near-best performance across six word similarity benchmarks (WordSim, MEN, SimLex-999, etc.), outperforming SGNS, CBOW, GloVe, and fastText.
• LCA with the flat window matrix was highly sensitive to window size, while the tail-cut kernel showed stable performance beyond window size 30, indicating reduced hyperparameter dependence.
• SCA, which uses labeled data (MEN and M.Turk) for supervision within the KCA framework, outperformed unsupervised LCA on most benchmarks, demonstrating the value of semi-supervised learning in KCA.
• The deterministic nature of CA ensures consistent results across runs, unlike Word2Vec, which produces different vectors due to random initialization.
• The theoretical link between Word2Vec and CA was validated: Word2Vec is shown to be a special case of KCA with a specific kernel and linear transformation.