[Paper Review] FRAGE: Frequency-Agnostic Word Representation

Chengyue Gong, Di He+4 more|arXiv (Cornell University)|Sep 18, 2018

Topic Modeling45 references79 citations

TL;DR

FRAGE introduces an adversarial training framework to learn frequency-agnostic word embeddings, reducing frequency bias and improving performance across word similarity, language modeling, machine translation, and text classification."

ABSTRACT

Continuous word representation (aka word embedding) is a basic building block in many neural network-based models used in natural language processing tasks. Although it is widely accepted that words with similar semantics should be close to each other in the embedding space, we find that word embeddings learned in several tasks are biased towards word frequency: the embeddings of high-frequency and low-frequency words lie in different subregions of the embedding space, and the embedding of a rare word and a popular word can be far from each other even if they are semantically similar. This makes learned word embeddings ineffective, especially for rare words, and consequently limits the performance of these neural network models. In this paper, we develop a neat, simple yet effective way to learn \emph{FRequency-AGnostic word Embedding} (FRAGE) using adversarial training. We conducted comprehensive studies on ten datasets across four natural language processing tasks, including word similarity, language modeling, machine translation and text classification. Results show that with FRAGE, we achieve higher performance than the baselines in all tasks.

Motivation & Objective

• Detect and characterize frequency-driven biases in standard word embeddings.
• Develop an adversarial training framework to make word representations less dependent on word frequency.
• Show that FRAGE improves downstream NLP task performance across multiple datasets and architectures.

Proposed method

• Split vocabulary into popular and rare words based on frequency.
• Introduce a discriminator that predicts whether an embedding belongs to a popular or rare word.
• Optimize a joint objective: L_T (task-specific loss) minus lambda times L_D (discriminator loss).
• Train model and embeddings to minimize task loss while fooling the discriminator, achieving frequency-agnostic embeddings.
• Discriminator is trained to maximize classification accuracy of popular vs. rare words; embeddings are trained to minimize task loss and fool the discriminator.

Experimental results

Research questions

• RQ1 Do word embeddings learned with FRAGE reduce frequency bias and better capture semantic relations for rare words?
• RQ2 Can FRAGE improve performance across diverse NLP tasks compared to task-specific baselines?
• RQ3 How does adversarial objective interact with standard task losses to yield more semantically meaningful embeddings?

Key findings

• FRAGE consistently outperforms baselines on word similarity tasks, with about 5.4 point improvement on the RW rare-word dataset.
• Language modeling gains observed: on PTB, improvements of 0.8, 1.2, 1.0 points in test perplexity for various checkpoints; on WT2, gains of 2.3, 2.4, 2.7 points.
• Machine translation shows BLEU gains: WMT14 En-De improvements of 1.06 (base) and 0.71 (big); IWSLT De-En gain of 0.85.
• Text classification improvements: accuracy gains of 1.26%, 0.66%, and 0.44% across AG’s News, IMDB, and 20NG datasets.
• Across 10 datasets spanning word similarity, language modeling, translation, and text classification, FRAGE demonstrates improved performance over baselines.