[论文解读] Learning to Compute Word Embeddings On the Fly

Dzmitry Bahdanau, Tom Bosc+4 more|arXiv (Cornell University)|Jun 1, 2017

Topic Modeling参考文献 23被引用 67

一句话总结

本文提出了基于辅助定义（例如字典定义、拼写）来预测即时嵌入以处理罕见词，从而提升任务性能并缩小与预训练嵌入的差距。

ABSTRACT

Words in natural language follow a Zipfian distribution whereby some words are frequent but most are rare. Learning representations for words in the "long tail" of this distribution requires enormous amounts of data. Representations of rare words trained directly on end tasks are usually poor, requiring us to pre-train embeddings on external data, or treat all rare words as out-of-vocabulary words with a unique representation. We provide a method for predicting embeddings of rare words on the fly from small amounts of auxiliary data with a network trained end-to-end for the downstream task. We show that this improves results against baselines where embeddings are trained on the end task for reading comprehension, recognizing textual entailment and language modeling.

研究动机与目标

• Motivate the rare-word problem caused by Zipfian word distributions and the limitations of fixed vocabularies.
• Propose a method to compute word embeddings from auxiliary data rather than maintaining a full set of embeddings.
• Demonstrate the approach on reading comprehension, textual entailment, and language modeling tasks.
• Show that auxiliary data approaches improve over end-task trained embeddings and close some gap to pretrained vectors.

提出的方法

• Define a definition reader that computes w-specific embeddings e_d(w) from auxiliary data d(w) (definitions or spellings).
• Combine e_d(w) with within-vocabulary embeddings e(w) as e_c(w)=e(w)+We_d(w) (or similar variants).
• Process multiple definitions in parallel and allow end-to-end training so semantic alignment with task-specific embeddings is preserved.
• Experiment with three definition encoders: mean pooling, mean pooling with a linear transform, and an LSTM reader for definitions; spelling uses an LSTM reader.
• Evaluate on SQuAD (reading comprehension), SNLI/MultiNLI (textual entailment), and OBW (language modeling) with and without GloVe pretraining.

实验结果

研究问题

• RQ1 Can on-the-fly embeddings computed from auxiliary data improve performance on NLP tasks compared to task-only learned embeddings?
• RQ2 How does dictionary-based and spelling-based auxiliary data contribute to representations of rare words?
• RQ3 To what extent can such auxiliary data bridge the gap to pretrained embeddings like GloVe across different tasks?
• RQ4 What are the practical effects on tasks with varying data regimes (SQuAD, SNLI/MultiNLI, language modeling)?

主要发现

• Auxiliary data improves performance over baselines trained only on the task objective in all experiments.
• Dictionary-based definitions (alone or with spelling) boost SQuAD EM, with SD outperforming S and approaching GloVe in development tests.
• Spelling information is especially beneficial for SQuAD, and combining dictionary and spelling (SD) yields the best results overall for reading comprehension.
• Dictionary definitions bridge roughly 40% of the gap between training-from-scratch and pretrained GloVe embeddings in SNLI and MultiNLI, with consistent gains on both datasets.
• In language modeling, adding spelling and dictionary information reduces perplexity and narrows the gap to GloVe, especially when the full training set is used.