[Paper Review] Learning to Compute Word Embeddings On the Fly

Dzmitry Bahdanau, Tom Bosc+4 more|arXiv (Cornell University)|Jun 1, 2017

Topic Modeling23 references67 citations

TL;DR

The paper introduces on-the-fly embeddings that are predicted from auxiliary definitions (e.g., dictionary definitions, spellings) to handle rare words, improving task performance and narrowing the gap with pretrained embeddings.

ABSTRACT

Words in natural language follow a Zipfian distribution whereby some words are frequent but most are rare. Learning representations for words in the "long tail" of this distribution requires enormous amounts of data. Representations of rare words trained directly on end tasks are usually poor, requiring us to pre-train embeddings on external data, or treat all rare words as out-of-vocabulary words with a unique representation. We provide a method for predicting embeddings of rare words on the fly from small amounts of auxiliary data with a network trained end-to-end for the downstream task. We show that this improves results against baselines where embeddings are trained on the end task for reading comprehension, recognizing textual entailment and language modeling.

Motivation & Objective

• Motivate the rare-word problem caused by Zipfian word distributions and the limitations of fixed vocabularies.
• Propose a method to compute word embeddings from auxiliary data rather than maintaining a full set of embeddings.
• Demonstrate the approach on reading comprehension, textual entailment, and language modeling tasks.
• Show that auxiliary data approaches improve over end-task trained embeddings and close some gap to pretrained vectors.

Proposed method

• Define a definition reader that computes w-specific embeddings e_d(w) from auxiliary data d(w) (definitions or spellings).
• Combine e_d(w) with within-vocabulary embeddings e(w) as e_c(w)=e(w)+We_d(w) (or similar variants).
• Process multiple definitions in parallel and allow end-to-end training so semantic alignment with task-specific embeddings is preserved.
• Experiment with three definition encoders: mean pooling, mean pooling with a linear transform, and an LSTM reader for definitions; spelling uses an LSTM reader.
• Evaluate on SQuAD (reading comprehension), SNLI/MultiNLI (textual entailment), and OBW (language modeling) with and without GloVe pretraining.

Experimental results

Research questions

• RQ1 Can on-the-fly embeddings computed from auxiliary data improve performance on NLP tasks compared to task-only learned embeddings?
• RQ2 How does dictionary-based and spelling-based auxiliary data contribute to representations of rare words?
• RQ3 To what extent can such auxiliary data bridge the gap to pretrained embeddings like GloVe across different tasks?
• RQ4 What are the practical effects on tasks with varying data regimes (SQuAD, SNLI/MultiNLI, language modeling)?

Key findings

• Auxiliary data improves performance over baselines trained only on the task objective in all experiments.
• Dictionary-based definitions (alone or with spelling) boost SQuAD EM, with SD outperforming S and approaching GloVe in development tests.
• Spelling information is especially beneficial for SQuAD, and combining dictionary and spelling (SD) yields the best results overall for reading comprehension.
• Dictionary definitions bridge roughly 40% of the gap between training-from-scratch and pretrained GloVe embeddings in SNLI and MultiNLI, with consistent gains on both datasets.
• In language modeling, adding spelling and dictionary information reduces perplexity and narrows the gap to GloVe, especially when the full training set is used.