[Paper Review] Hierarchical Text Classification of Urdu News using Deep Neural Network

Taimoor Ahmed Javed, Waseem Shahzad+1 more|arXiv (Cornell University)|Jul 7, 2021

Text and Document Classification Technologies28 references4 citations

TL;DR

This paper proposes a Hierarchical Multi-Layer LSTM (HMLSTM) model for deep learning-based hierarchical text classification of Urdu news, using a novel Urdu-specific hierarchical LSTM layer and Word2Vec embeddings. It achieves state-of-the-art performance on a newly created dataset, outperforming traditional models and CNNs with a Micro-F1 of 0.9683 and Macro-F1 of 0.8927.

ABSTRACT

Digital text is increasing day by day on the internet. It is very challenging to classify a large and heterogeneous collection of data, which require improved information processing methods to organize text. To classify large size of corpus, one common approach is to use hierarchical text classification, which aims to classify textual data in a hierarchical structure. Several approaches have been proposed to tackle classification of text but most of the research has been done on English language. This paper proposes a deep learning model for hierarchical text classification of news in Urdu language - consisting of 51,325 sentences from 8 online news websites belonging to the following genres: Sports; Technology; and Entertainment. The objectives of this paper are twofold: (1) to develop a large human-annotated dataset of news in Urdu language for hierarchical text classification; and (2) to classify Urdu news hierarchically using our proposed model based on LSTM mechanism named as Hierarchical Multi-layer LSTMs (HMLSTM). Our model consists of two modules: Text Representing Layer, for obtaining text representation in which we use Word2vec embedding to transform the words to vector and Urdu Hierarchical LSTM Layer (UHLSTML) an end-to-end fully connected deep LSTMs network to perform automatic feature learning, we train one LSTM layer for each level of the class hierarchy. We have performed extensive experiments on our self created dataset named as Urdu News Dataset for Hierarchical Text Classification (UNDHTC). The result shows that our proposed method is very effective for hierarchical text classification and it outperforms baseline methods significantly and also achieved good results as compare to deep neural model.

Motivation & Objective

• To develop a large-scale, human-annotated dataset for hierarchical text classification in Urdu, named Urdu News Dataset for Hierarchical Text Classification (UNDHTC), comprising 51,325 sentences across 12 classes in three genres: Sports, Technology, and Entertainment.
• To address the lack of research in hierarchical text classification for Urdu, a low-resource language with complex morphology and limited linguistic resources.
• To design and implement a deep neural network model capable of learning hierarchical representations from raw Urdu text for improved classification accuracy.
• To evaluate the model's performance against multiple baseline machine learning and deep learning models on the proposed dataset.
• To demonstrate the effectiveness of end-to-end hierarchical learning using LSTMs in capturing long-range dependencies and hierarchical label structures in Urdu text.

Proposed method

• Employ Word2Vec embeddings to convert Urdu words into dense vector representations in a continuous vector space.
• Design a novel Urdu Hierarchical LSTM Layer (UHLSTML), a fully connected deep LSTM network that processes text at each level of the hierarchical class structure.
• Train one separate LSTM layer for each level of the class hierarchy to learn hierarchical features and dependencies between parent and child categories.
• Implement an end-to-end training pipeline where the Text Representing Layer feeds into the UHLSTML for joint optimization of text and label representations.
• Use a hierarchical multi-label classification framework where predictions are made at each level and propagated through the hierarchy.
• Apply standard evaluation metrics including Micro-F1 and Macro-F1 to assess performance across all classes and levels.

Experimental results

Research questions

• RQ1 Can a deep neural network model effectively learn hierarchical representations for Urdu text classification, given the language's morphological complexity and limited resources?
• RQ2 How does the proposed HMLSTM model compare to traditional machine learning models (e.g., SVM, Random Forest) and deep learning baselines (e.g., CNN) in classifying Urdu news hierarchically?
• RQ3 To what extent does the hierarchical structure of the model improve classification performance compared to flat classification approaches?
• RQ4 How well does the model generalize across different genres (Sports, Technology, Entertainment) in the Urdu news domain?
• RQ5 What are the limitations of the model in handling ambiguous or contextually misleading keywords (e.g., 'actor' in entertainment vs. sports)?

Key findings

• The HMLSTM model achieved a Micro-F1 score of 0.9683, significantly outperforming all baseline models, including SVM (0.8878) and Logistic Regression (0.8980).
• HMLSTM achieved a Macro-F1 of 0.8927, indicating strong performance across all individual classes, especially in fine-grained subcategories.
• The model outperformed CNN (Micro-F1: 0.9323) and all traditional machine learning models, demonstrating the effectiveness of hierarchical LSTM for Urdu text.
• Qualitative analysis showed that HMLSTM correctly classified long and complex sentences due to its long-term memory retention via LSTM units.
• The model struggled with contextually ambiguous terms (e.g., 'actor' in entertainment news), indicating a need for improved contextual modeling in future work.
• The results confirm that hierarchical modeling via multi-level LSTMs enables better capture of label correlations and structural dependencies than flat classification methods.