[論文レビュー] On the Dimensionality of Word Embedding

Zi Yin, Yuanyuan Shen|arXiv (Cornell University)|Dec 11, 2018

Topic Modeling参考文献 41被引用数 116

ひとこと要約

PIP損失を導入する、単位系不変な指標を用いて埋め込み次元が単語ベクトルに与える影響を分析し、バイアス-分散のトレードオフを明らかにし、LSA、Word2Vec skip-gram、GloVeの最適な次元を選択するための principled な手法を提供する。

ABSTRACT

In this paper, we provide a theoretical understanding of word embedding and its dimensionality. Motivated by the unitary-invariance of word embedding, we propose the Pairwise Inner Product (PIP) loss, a novel metric on the dissimilarity between word embeddings. Using techniques from matrix perturbation theory, we reveal a fundamental bias-variance trade-off in dimensionality selection for word embeddings. This bias-variance trade-off sheds light on many empirical observations which were previously unexplained, for example the existence of an optimal dimensionality. Moreover, new insights and discoveries, like when and how word embeddings are robust to over-fitting, are revealed. By optimizing over the bias-variance trade-off of the PIP loss, we can explicitly answer the open question of dimensionality selection for word embedding.

研究の動機と目的

• Explain the dimensionality problem in word embeddings and motivate the need for a universal criterion.
• Introduce a unitary-invariant loss (PIP loss) for embeddings and connect it to downstream functionality.
• Develop a bias-variance framework to characterize dimensionality effects using matrix perturbation theory.
• Provide a practical procedure to select optimal embedding dimensionality by minimizing PIP loss across algorithms (LSA, Word2Vec, GloVe).

提案手法

• Define PIP matrix as EE^T to capture pairwise inner products.
• Prove PIP loss is unitary-invariant and respects embedding functionality.
• Derive bias-variance decomposition of PIP loss in special case (alpha=0) and general case (alpha in (0,1]).
• Apply perturbation theory to bound PIP loss and reveal an optimal dimensionality k* balancing signal preservation and noise.
• Propose Monte Carlo and spectrum estimation (USVT) approaches to estimate spectrum and noise for dimensionality selection.
• Validate by experiments on Text8 corpus across LSA, skip-gram Word2Vec, and GloVe, comparing theoretical k* to empirical performance.

実験結果

リサーチクエスチョン

• RQ1 What is a unitary-invariant metric that quantifies similarity/dissimilarity of word embeddings across coordinate systems?
• RQ2 How does embedding dimensionality influence the quality of embeddings under a bias-variance perspective?
• RQ3 Can we quantify robustness to overfitting of embedding methods via a parameter alpha in the factorization, and what does this imply for popular methods (Word2Vec, GloVe)?
• RQ4 Can we explicitly determine an optimal embedding dimensionality by minimizing a principled loss (PIP loss) and validate it empirically?
• RQ5 How can spectrum and noise estimation be used to select dimensionality for LSA, Word2Vec, and GloVe?

主な発見

• A Pairwise Inner Product (PIP) loss is a unitary-invariant metric suitable for evaluating embeddings.
• There is a fundamental bias-variance trade-off in dimensionality selection, yielding an optimal dimensionality.
• Embedding robustness to over-fitting increases with the exponent alpha in the factorization; skip-gram and GloVe (alpha ≈ 0.5) are robust to over-parameterization.
• Minimizing PIP loss provides a principled solution to dimensionality selection, demonstrated for LSA, Word2Vec, and GloVe on Text8.
• Monte Carlo and spectrum-noise estimation methods can accurately approximate the PIP loss and guide k* selection.
• Empirical results show k* from PIP loss aligns with optimal dimensionalities in intrinsic word relatedness and analogy tests.