[Paper Review] Generalizing Word Embeddings using Bag of Subwords

Jinman Zhao, Sidharth Mudgal+1 more|arXiv (Cornell University)|Sep 12, 2018

Natural Language Processing Techniques4 citations

TL;DR

This paper proposes a subword-level word embedding model called Bag-of-Substring (BoS) that generalizes pre-trained word vectors to out-of-vocabulary (OOV) words by treating words as bags of character n-grams. The method learns subword-vector relationships from a fixed vocabulary and achieves state-of-the-art performance in word similarity and joint POS/morphosyntactic tagging across 23 languages, demonstrating effective morphological generalization without context.

ABSTRACT

We approach the problem of generalizing pre-trained word embeddings beyond fixed-size vocabularies without using additional contextual information. We propose a subword-level word vector generation model that views words as bags of character $n$-grams. The model is simple, fast to train and provides good vectors for rare or unseen words. Experiments show that our model achieves state-of-the-art performances in English word similarity task and in joint prediction of part-of-speech tag and morphosyntactic attributes in 23 languages, suggesting our model's ability in capturing the relationship between words' textual representations and their embeddings.

Motivation & Objective

• To address the out-of-vocabulary (OOV) word problem in fixed-vocabulary word embeddings by generalizing pre-trained vectors to rare or unseen words.
• To develop a method that infers word vectors for OOV words using only the word's lexical form and pre-trained vectors, without relying on contextual information.
• To capture morphological and syntactic features of words through subword components, enabling consistent vector estimation for rare or agglutinative language words.
• To evaluate the model’s ability to generalize semantic and grammatical features across diverse languages, especially in low-resource or morphologically rich settings.
• To provide a simple, efficient, and context-free alternative to existing subword or RNN-based OOV embedding methods.

Proposed method

• The model treats each word as a bag of character n-grams (substrings of length n), representing words as collections of subword units.
• It learns a mapping from subword n-grams to word vectors using a lookup table trained on in-vocabulary words from a pre-trained embedding set (e.g., Polyglot).
• During inference, the model aggregates the learned subword vectors of an OOV word to produce its final word vector via averaging or weighted combination.
• The method is trained end-to-end using a loss function that minimizes the difference between predicted and true word vectors for in-vocabulary words.
• The model is designed to be fast and scalable, avoiding recurrent or attention-based architectures to maintain efficiency.
• It generalizes to OOV words by leveraging subword compositionality, assuming that morphological structure encodes semantic and syntactic information.

Experimental results

Research questions

• RQ1 Can a subword-level model generalize pre-trained word embeddings to OOV words without using contextual information?
• RQ2 How well can a bag-of-subwords approach capture morphological and syntactic relationships in words compared to existing subword or RNN-based methods?
• RQ3 Does the model achieve state-of-the-art performance in intrinsic (word similarity) and extrinsic (POS tagging) evaluations across diverse languages?
• RQ4 To what extent does the model’s performance depend on morphological richness or language family characteristics?
• RQ5 Can the model generalize consistently across languages with varying degrees of agglutination and inflectional complexity?

Key findings

• The BoS model achieves state-of-the-art performance on the English word similarity task, outperforming prior subword-level models under the same setting.
• In joint POS tagging and morphosyntactic attribute prediction across 23 languages, BoS achieves higher micro F1 scores than MIMICK and random baselines, with gains up to 0.125 F1 points in some languages.
• The model shows particularly strong gains in agglutinative languages like Turkish, Indonesian, and Finnish, indicating effective capture of morphological structure.
• For Germanic and Romance languages such as English, Swedish, and Spanish, the model consistently outperforms MIMICK, with average F1 improvements of 0.03–0.06 across tasks.
• The model achieves a 0.947 micro F1 score on the English POS tagging task, outperforming MIMICK by 0.089 F1 points in the same evaluation setup.
• The method generalizes effectively to unseen words, demonstrating that subword compositionality can encode linguistic knowledge even without context.