[Paper Review] Unsupervised Morphological Expansion of Small Datasets for Improving Word Embeddings

Syed Sarfaraz Akhtar, Arihant Gupta+3 more|arXiv (Cornell University)|Nov 15, 2017

Topic Modeling18 references3 citations

TL;DR

This paper proposes an unsupervised, language-independent method to improve word embeddings by generating artificial sentences through morphological expansion of low-resource datasets. By training on these synthetic sentences, the model enhances embeddings for rare and low-frequency words, achieving performance comparable to models trained on much larger corpora, especially in morphologically rich languages like Hindi and German.

ABSTRACT

We present a language independent, unsupervised method for building word embeddings using morphological expansion of text. Our model handles the problem of data sparsity and yields improved word embeddings by relying on training word embeddings on artificially generated sentences. We evaluate our method using small sized training sets on eleven test sets for the word similarity task across seven languages. Further, for English, we evaluated the impacts of our approach using a large training set on three standard test sets. Our method improved results across all languages.

Motivation & Objective

• Address data sparsity in training word embeddings for low-resource and morphologically rich languages.
• Improve word embeddings for rare and unseen words without relying on external morphological analyzers.
• Develop a language-agnostic, unsupervised technique that generates artificial training sentences using morphological rules.
• Demonstrate that morphological expansion can yield embeddings comparable to those trained on significantly larger datasets.
• Release a new Hindi word similarity dataset (Hin-WS235) to support evaluation in low-resource settings.

Proposed method

• Apply unsupervised morphological expansion to generate artificial sentences by applying morphological transformations (e.g., adding suffixes, prefixes) to words in the training corpus.
• Use the Skip-gram model with negative sampling and fixed hyperparameters (d=500, min_count=5) for training word embeddings.
• Implement a hierarchical word lookup strategy ('Morph' step) to recover embeddings for rare or unseen words by searching for morphological variants with higher frequency.
• Apply capitalization and length-based heuristics to improve embedding recovery when direct lookup fails.
• Use frequency and length as tiebreakers when multiple morphological variants are available.
• Combine morphological expansion with word embedding training to improve representations for low-frequency words.

Experimental results

Research questions

• RQ1 Can morphological expansion of small datasets improve word embedding quality across multiple languages, especially morphologically rich ones?
• RQ2 How does the proposed method compare to state-of-the-art models trained on much larger corpora in terms of word similarity performance?
• RQ3 Does the use of artificial sentences generated via morphological rules mitigate data sparsity for rare and unseen words?
• RQ4 Why does the method perform less effectively on Arabic, and what linguistic factors contribute to this limitation?
• RQ5 To what extent does morphological expansion preserve analogical regularities in the vector space?

Key findings

• The method significantly improves word similarity performance on small datasets across seven languages, with the largest gains observed in morphologically rich languages like Hindi and German.
• For English, the model achieves 47.9 on the Stanford Rare-Word (RW) dataset with SG + Exp + Morph, outperforming the baseline SG model (42.1) and matching results from models trained on 42B tokens.
• On the RG65 dataset, the method achieves 80.4 accuracy with SG + Exp + Morph, outperforming GloVe (82.9) when trained on 42B tokens but using only 1B tokens.
• The method performs worse on the WS353 dataset, where accuracy slightly decreases with expansion, likely due to disruption of analogical regularities by grammatically incorrect artificial sentences.
• The model shows robustness in recovering embeddings for rare words via morphological variant lookup, especially when frequency and length heuristics are applied.
• The approach is less effective on Arabic, attributed to the language's complex infixation patterns that are poorly handled by the current prefix/suffix-based expansion strategy.