[Paper Review] Dirichlet-Smoothed Word Embeddings for Low-Resource Settings

Jakob Jungmaier, Nora Kassner+1 more|arXiv (Cornell University)|Jun 22, 2020

Topic Modeling7 references5 citations

TL;DR

This paper proposes Dirichlet-smoothed PPMI word embeddings to mitigate the bias of standard PPMI toward rare words in low-resource settings. By applying Dirichlet smoothing to co-occurrence matrices before truncated SVD, the method achieves superior performance on small corpora and competitive results on low-resource languages like Maltese and Luxembourgish, outperforming both word2vec and PU-Learning in low-data regimes.

ABSTRACT

Nowadays, classical count-based word embeddings using positive pointwise mutual information (PPMI) weighted co-occurrence matrices have been widely superseded by machine-learning-based methods like word2vec and GloVe. But these methods are usually applied using very large amounts of text data. In many cases, however, there is not much text data available, for example for specific domains or low-resource languages. This paper revisits PPMI by adding Dirichlet smoothing to correct its bias towards rare words. We evaluate on standard word similarity data sets and compare to word2vec and the recent state of the art for low-resource settings: Positive and Unlabeled (PU) Learning for word embeddings. The proposed method outperforms PU-Learning for low-resource settings and obtains competitive results for Maltese and Luxembourgish.

Motivation & Objective

• To address the well-known bias of PPMI toward rare words in low-resource NLP settings.
• To evaluate whether classical count-based word embedding methods can outperform modern machine learning approaches like word2vec and GloVe in low-data regimes.
• To investigate the effectiveness of Dirichlet smoothing in improving PPMI-based embeddings for rare and low-frequency words.
• To demonstrate practical applicability on low-resource languages such as Maltese and Luxembourgish using Wikipedia dumps.
• To provide a publicly available implementation for reproducibility and further research.

Proposed method

• The method applies Dirichlet smoothing to the co-occurrence matrix before computing PPMI, using a smoothing parameter λ to control the degree of smoothing.
• The smoothed co-occurrence probabilities are used to compute a modified PMI score, which reduces overestimation of rare word co-occurrences.
• The resulting smoothed matrix is then processed via truncated singular value decomposition (SVD) to obtain dense word embeddings.
• The optimal λ value is selected per corpus size by evaluating performance on held-out similarity tasks, with values derived from larger context windows.
• The approach uses a symmetric windowing scheme to collect word co-occurrences and applies maximum likelihood estimation for probability estimation.
• For low-resource languages, λ values are transferred from similar-sized monolingual corpora (e.g., enwik9) to ensure robustness.

Experimental results

Research questions

• RQ1 Can Dirichlet-smoothed PPMI embeddings outperform state-of-the-art methods like PU-Learning and word2vec in low-resource settings?
• RQ2 How does Dirichlet smoothing mitigate the bias of PPMI toward rare words in small corpora?
• RQ3 Does the proposed method achieve competitive performance on rare word similarity tasks compared to existing approaches?
• RQ4 To what extent can the method generalize to low-resource and endangered languages like Maltese and Luxembourgish?
• RQ5 How sensitive is the performance to the choice of the smoothing parameter λ, and how can it be optimized efficiently?

Key findings

• SVD-PPMI with Dirichlet smoothing (SVD-PPMI λ) outperforms both word2vec SGNS and PU-Learning on the first 1M and 2M tokens of enwik9, demonstrating superior performance in low-data regimes.
• The method achieves the highest performance on the RW (Rare Word) similarity dataset, indicating strong effectiveness for rare word representation.
• For Maltese and Luxembourgish, the method obtains competitive results, outperforming PU-Learning and showing promise for low-resource language modeling.
• Performance improves with increasing corpus size, but the advantage of SVD-PPMI λ is most pronounced on smaller corpora, where data sparsity is most problematic.
• The optimal λ value increases with corpus size, and the method shows robustness by transferring λ values from larger enwik9 segments to low-resource language models.
• The authors release their code publicly, enabling reproducibility and extension to other low-resource settings.