[논문 리뷰] On the Dimensionality of Word Embedding

Zi Yin, Yuanyuan Shen|arXiv (Cornell University)|2018. 12. 11.

Topic Modeling참고 문헌 41인용 수 116

한 줄 요약

PIP loss를 도입하여 단위 불변량 메트릭으로 임베딩 차원이 단어 벡터에 미치는 영향을 분석하고, 편향-분산 트레이드오프를 드러내며, LSA, Word2Vec skip-gram, GloVe에 대해 최적 차원 수를 선택하는 원칙적 방법을 제시한다.

ABSTRACT

In this paper, we provide a theoretical understanding of word embedding and its dimensionality. Motivated by the unitary-invariance of word embedding, we propose the Pairwise Inner Product (PIP) loss, a novel metric on the dissimilarity between word embeddings. Using techniques from matrix perturbation theory, we reveal a fundamental bias-variance trade-off in dimensionality selection for word embeddings. This bias-variance trade-off sheds light on many empirical observations which were previously unexplained, for example the existence of an optimal dimensionality. Moreover, new insights and discoveries, like when and how word embeddings are robust to over-fitting, are revealed. By optimizing over the bias-variance trade-off of the PIP loss, we can explicitly answer the open question of dimensionality selection for word embedding.

연구 동기 및 목표

• Explain the dimensionality problem in word embeddings and motivate the need for a universal criterion.
• Introduce a unitary-invariant loss (PIP loss) for embeddings and connect it to downstream functionality.
• Develop a bias-variance framework to characterize dimensionality effects using matrix perturbation theory.
• Provide a practical procedure to select optimal embedding dimensionality by minimizing PIP loss across algorithms (LSA, Word2Vec, GloVe).

제안 방법

• Define PIP matrix as EE^T to capture pairwise inner products.
• Prove PIP loss is unitary-invariant and respects embedding functionality.
• Derive bias-variance decomposition of PIP loss in special case (alpha=0) and general case (alpha in (0,1]).
• Apply perturbation theory to bound PIP loss and reveal an optimal dimensionality k* balancing signal preservation and noise.
• Propose Monte Carlo and spectrum estimation (USVT) approaches to estimate spectrum and noise for dimensionality selection.
• Validate by experiments on Text8 corpus across LSA, skip-gram Word2Vec, and GloVe, comparing theoretical k* to empirical performance.

실험 결과

연구 질문

• RQ1 What is a unitary-invariant metric that quantifies similarity/dissimilarity of word embeddings across coordinate systems?
• RQ2 How does embedding dimensionality influence the quality of embeddings under a bias-variance perspective?
• RQ3 Can we quantify robustness to overfitting of embedding methods via a parameter alpha in the factorization, and what does this imply for popular methods (Word2Vec, GloVe)?
• RQ4 Can we explicitly determine an optimal embedding dimensionality by minimizing a principled loss (PIP loss) and validate it empirically?
• RQ5 How can spectrum and noise estimation be used to select dimensionality for LSA, Word2Vec, and GloVe?

주요 결과

• A Pairwise Inner Product (PIP) loss is a unitary-invariant metric suitable for evaluating embeddings.
• There is a fundamental bias-variance trade-off in dimensionality selection, yielding an optimal dimensionality.
• Embedding robustness to over-fitting increases with the exponent alpha in the factorization; skip-gram and GloVe (alpha ≈ 0.5) are robust to over-parameterization.
• Minimizing PIP loss provides a principled solution to dimensionality selection, demonstrated for LSA, Word2Vec, and GloVe on Text8.
• Monte Carlo and spectrum-noise estimation methods can accurately approximate the PIP loss and guide k* selection.
• Empirical results show k* from PIP loss aligns with optimal dimensionalities in intrinsic word relatedness and analogy tests.