[논문 리뷰] FRAGE: Frequency-Agnostic Word Representation

Chengyue Gong, Di He외 4명|arXiv (Cornell University)|2018. 09. 18.

Topic Modeling참고 문헌 45인용 수 79

한 줄 요약

FRAGE는 빈도 비편향 단어 임베딩을 학습하기 위한 적대적 학습 프레임워크를 도입하여 빈도 편향을 줄이고 단어 유사도, 언어 모델링, 기계 번역, 텍스트 분류 전반에서 성능을 향상시킨다.

ABSTRACT

Continuous word representation (aka word embedding) is a basic building block in many neural network-based models used in natural language processing tasks. Although it is widely accepted that words with similar semantics should be close to each other in the embedding space, we find that word embeddings learned in several tasks are biased towards word frequency: the embeddings of high-frequency and low-frequency words lie in different subregions of the embedding space, and the embedding of a rare word and a popular word can be far from each other even if they are semantically similar. This makes learned word embeddings ineffective, especially for rare words, and consequently limits the performance of these neural network models. In this paper, we develop a neat, simple yet effective way to learn \emph{FRequency-AGnostic word Embedding} (FRAGE) using adversarial training. We conducted comprehensive studies on ten datasets across four natural language processing tasks, including word similarity, language modeling, machine translation and text classification. Results show that with FRAGE, we achieve higher performance than the baselines in all tasks.

연구 동기 및 목표

• Detect and characterize frequency-driven biases in standard word embeddings.
• Develop an adversarial training framework to make word representations less dependent on word frequency.
• Show that FRAGE improves downstream NLP task performance across multiple datasets and architectures.

제안 방법

• Split vocabulary into popular and rare words based on frequency.
• Introduce a discriminator that predicts whether an embedding belongs to a popular or rare word.
• Optimize a joint objective: L_T (task-specific loss) minus lambda times L_D (discriminator loss).
• Train model and embeddings to minimize task loss while fooling the discriminator, achieving frequency-agnostic embeddings.
• Discriminator is trained to maximize classification accuracy of popular vs. rare words; embeddings are trained to minimize task loss and fool the discriminator.

실험 결과

연구 질문

• RQ1 Do word embeddings learned with FRAGE reduce frequency bias and better capture semantic relations for rare words?
• RQ2 Can FRAGE improve performance across diverse NLP tasks compared to task-specific baselines?
• RQ3 How does adversarial objective interact with standard task losses to yield more semantically meaningful embeddings?

주요 결과

• FRAGE consistently outperforms baselines on word similarity tasks, with about 5.4 point improvement on the RW rare-word dataset.
• Language modeling gains observed: on PTB, improvements of 0.8, 1.2, 1.0 points in test perplexity for various checkpoints; on WT2, gains of 2.3, 2.4, 2.7 points.
• Machine translation shows BLEU gains: WMT14 En-De improvements of 1.06 (base) and 0.71 (big); IWSLT De-En gain of 0.85.
• Text classification improvements: accuracy gains of 1.26%, 0.66%, and 0.44% across AG’s News, IMDB, and 20NG datasets.
• Across 10 datasets spanning word similarity, language modeling, translation, and text classification, FRAGE demonstrates improved performance over baselines.