[Paper Review] Text classification dataset and analysis for Uzbek language

Elmurod Kuriyozov, Ulugbek Salaev+2 more|arXiv (Cornell University)|Feb 28, 2023

Text and Document Classification Technologies4 citations

TL;DR

This paper introduces a new multi-label text classification dataset for the Uzbek language, compiled from 10 news and press websites across 15 categories. It evaluates traditional and deep learning models, demonstrating that transformer-based BERTbek outperforms RNN, CNN, and rule-based models, establishing a strong baseline for future Uzbek NLP research.

ABSTRACT

Text classification is an important task in Natural Language Processing (NLP), where the goal is to categorize text data into predefined classes. In this study, we analyse the dataset creation steps and evaluation techniques of multi-label news categorisation task as part of text classification. We first present a newly obtained dataset for Uzbek text classification, which was collected from 10 different news and press websites and covers 15 categories of news, press and law texts. We also present a comprehensive evaluation of different models, ranging from traditional bag-of-words models to deep learning architectures, on this newly created dataset. Our experiments show that the Recurrent Neural Network (RNN) and Convolutional Neural Network (CNN) based models outperform the rule-based models. The best performance is achieved by the BERTbek model, which is a transformer-based BERT model trained on the Uzbek corpus. Our findings provide a good baseline for further research in Uzbek text classification.

Motivation & Objective

• To create a large-scale, multi-label text classification dataset for the Uzbek language to support NLP research.
• To evaluate the performance of diverse models—ranging from bag-of-words to deep learning—on Uzbek text classification.
• To establish a benchmark for future research in low-resource Uzbek NLP by comparing rule-based, RNN, CNN, and transformer-based models.
• To analyze dataset construction techniques and evaluation protocols for multi-label news categorization in Uzbek.

Proposed method

• Collected text data from 10 diverse Uzbek news and press websites to ensure broad coverage of topics and styles.
• Annotated the dataset with 15 predefined categories, including news, press releases, and legal texts, for multi-label classification.
• Implemented and evaluated multiple models: rule-based classifiers, traditional bag-of-words with SVM, RNNs, CNNs, and the BERTbek model fine-tuned on a Uzbek corpus.
• Applied standard NLP preprocessing steps, including tokenization and sequence padding, to prepare data for deep learning models.
• Used standard evaluation metrics such as macro-F1, micro-F1, and accuracy to compare model performance across categories.
• Fine-tuned the BERTbek model—a multilingual BERT variant pre-trained on Uzbek text—to optimize for the multi-label classification task.

Experimental results

Research questions

• RQ1 What is the performance of rule-based and traditional machine learning models on Uzbek text classification?
• RQ2 How do RNN and CNN architectures compare in classifying multi-label Uzbek text?
• RQ3 To what extent does a BERT-based model fine-tuned on Uzbek data improve classification accuracy over non-transformer models?
• RQ4 What are the key challenges in constructing a high-quality, multi-label dataset for low-resource languages like Uzbek?

Key findings

• The BERTbek model achieved the highest performance, outperforming all other models, including RNN and CNN, on the Uzbek text classification dataset.
• RNN and CNN models showed significant improvements over rule-based baselines, indicating the value of deep learning for Uzbek NLP.
• The multi-label dataset spans 15 distinct categories and includes diverse content from news, press, and legal sources, enhancing its representativeness.
• The study establishes a new benchmark for Uzbek text classification, with BERTbek setting a strong performance baseline for future research.