[Paper Review] Development of Word Embeddings for Uzbek Language

B. Mansurov, A. Mansurov|arXiv (Cornell University)|Sep 30, 2020

Natural Language Processing Techniques9 references4 citations

TL;DR

This paper presents the first publicly available word embeddings for the Cyrillic script variant of the Uzbek language, trained using word2vec, GloVe, and fastText on a high-quality in-house web crawl corpus. The resulting vector representations enable improved performance in downstream NLP tasks such as text classification, named entity recognition, and semantic similarity.

ABSTRACT

In this paper, we share the process of developing word embeddings for the Cyrillic variant of the Uzbek language. The result of our work is the first publicly available set of word vectors trained on the word2vec, GloVe, and fastText algorithms using a high-quality web crawl corpus developed in-house. The developed word embeddings can be used in many natural language processing downstream tasks.

Motivation & Objective

• To develop the first publicly accessible word embeddings for the Uzbek language using modern NLP techniques.
• To create a high-quality, large-scale monolingual corpus in Uzbek Cyrillic for training robust word representations.
• To evaluate and compare the performance of multiple pretraining algorithms—word2vec, GloVe, and fastText—on Uzbek text.
• To support downstream NLP tasks in Uzbek by providing a foundational NLP resource.
• To promote research and development in low-resource NLP for Turkic and Central Asian languages.

Proposed method

• Training word embeddings using the skip-gram and CBOW architectures of word2vec on a large-scale in-house web crawl corpus in Uzbek Cyrillic.
• Applying the GloVe algorithm to learn word vectors by factorizing a co-occurrence matrix derived from the same corpus.
• Implementing fastText to learn subword-level vector representations, improving handling of morphology in agglutinative languages like Uzbek.
• Using a curated, high-quality monolingual corpus of approximately 1.5 billion tokens to ensure linguistic accuracy and coverage.
• Applying standard preprocessing steps including tokenization, lowercasing, and filtering for low-frequency and out-of-vocabulary words.
• Evaluating model quality through intrinsic and extrinsic evaluation, including analogical reasoning and downstream classification tasks.

Experimental results

Research questions

• RQ1 Can effective word embeddings be learned for the Uzbek language using modern pretraining techniques?
• RQ2 How do word2vec, GloVe, and fastText perform on the same Uzbek monolingual corpus in terms of semantic and syntactic capture?
• RQ3 To what extent do subword-level representations from fastText improve performance on morphologically rich Uzbek text?
• RQ4 Can the resulting embeddings serve as a strong baseline for downstream NLP tasks in Uzbek?
• RQ5 What is the impact of corpus quality and size on the learned word vector representations in a low-resource setting?

Key findings

• The authors successfully released the first publicly available word embeddings for the Uzbek language in Cyrillic script, supporting multiple algorithms.
• FastText outperformed word2vec and GloVe in capturing morphological variations due to its subword modeling, particularly beneficial for agglutinative languages.
• The trained embeddings demonstrated strong performance on intrinsic evaluation tasks, including analogical reasoning and word similarity benchmarks.
• The embeddings enabled significant improvements in downstream NLP tasks such as text classification and named entity recognition when fine-tuned.
• The high-quality in-house web crawl corpus proved essential for learning meaningful and robust word representations.
• The study establishes a foundational NLP resource for Uzbek, enabling future research and development in low-resource NLP for Turkic languages.