[論文レビュー] Tying Word Vectors and Word Classifiers: A Loss Framework for Language Modeling

Hakan Inan, Khashayar Khosravi+1 more|arXiv (Cornell University)|Nov 4, 2016

Topic Modeling参考文献 17被引用数 173

ひとこと要約

本論文は、語彙埋め込みの類似性に基づくKL-divergence項をクロスエントロピーに組み込んだ言語モデリングの損失フレームワークを提案し、入力埋め込みを出力射影として再利用することで性能を著しく向上させ、パラメータを削減できることを示す。

ABSTRACT

Recurrent neural networks have been very successful at predicting sequences of words in tasks such as language modeling. However, all such models are based on the conventional classification framework, where the model is trained against one-hot targets, and each word is represented both as an input and as an output in isolation. This causes inefficiencies in learning both in terms of utilizing all of the information and in terms of the number of parameters needed to train. We introduce a novel theoretical framework that facilitates better learning in language modeling, and show that our framework leads to tying together the input embedding and the output projection matrices, greatly reducing the number of trainable variables. Our framework leads to state of the art performance on the Penn Treebank with a variety of network models.

研究の動機と目的

• Identify inefficiencies in standard one-hot target language modeling frameworks where inputs and outputs live in the same space but are learned separately.
• Introduce a KL-divergence based augmentation to the cross-entropy loss using a word-embedding informed target distribution.
• Theoretically justify and empirically validate reusing the input embedding matrix as the output projection to reduce parameters while preserving performance.
• Demonstrate improvements over conventional RNNLMs on Penn Treebank and Wikitext-2 across network sizes.

提案手法

• Define augmented loss J^tot = J + alpha J^aug where J is cross-entropy and J^aug is KL(ỹ || ŷ) with temperature tau.
• Construct ŷ from the network logits and ỹ from embedding-based similarities: ŷ uses softmax(Wh_t / tau) and ỹ uses softmax(L^T u_t / tau) where u_t = L y*_t.
• Theoretically analyze the high-temperature regime showing W h_t ≈ L^T u_t, implying the output projection aligns with the embedding matrix.
• Propose a practical scheme to either (i) reuse the embedding matrix by setting W ≈ L^T (and b = 0) or (ii) combine with augmented loss for improved learning.
• Experiment with two datasets (PTB, Wikitext-2) and three model scales (small/medium/large LSTMs) under variational dropout.

実験結果

リサーチクエスチョン

• RQ1 Does augmenting cross-entropy with an embedding-based KL loss improve language model performance on standard benchmarks?
• RQ2 Does sharing the input embedding with the output projection (W ≈ L^T) reduce parameters and maintain or improve perplexity?
• RQ3 How do the augmentation (AL), embedding reuse (RE), and their combination (REAL) compare across dataset sizes and model capacities?
• RQ4 Is the improved learning mechanism consistent across PTB and Wikitext-2 datasets?

主な発見

• Augmented loss and embedding reuse both outperform the baseline VD-LSTM across PTB and Wikitext-2.
• Using augmented loss (AL) yields strong gains for smaller networks, with larger gains on PTB than on the larger Wikitext-2 dataset.
• Embedding reuse (RE) provides substantial improvements, especially for larger networks, and largely accounts for parameter reduction.
• The combination (REAL) achieves the best overall perplexities, often surpassing prior state-of-the-art results on PTB.
• Empirical results show the proposed framework reduces the subspace distance between L^T and W, supporting the theoretical justification for embedding reuse.
• Qualitative analysis indicates more accurate target word probabilities and fewer unwarranted predictions like </unk> when using the proposed framework.