[Paper Review] LA-HCN: Label-based Attention for Hierarchical Multi-label TextClassification Neural Network

Xinyi Zhang, Jiahao Xu+2 more|arXiv (Cornell University)|Sep 23, 2020

Text and Document Classification Technologies26 references4 citations

TL;DR

This paper proposes LA-HCN, a novel neural network for hierarchical multi-label text classification that uses label-based attention to extract task-specific features at each level of the label hierarchy. By learning hierarchical representations through component embeddings and employing separate local and global document embeddings, LA-HCN achieves state-of-the-art performance across four benchmark datasets while enabling interpretable attention visualization tied to specific labels.

ABSTRACT

Hierarchical multi-label text classification (HMTC) has been gaining popularity in recent years thanks to its applicability to a plethora of real-world applications. The existing HMTC algorithms largely focus on the design of classifiers, such as the local, global, or a combination of them. However, very few studies have focused on hierarchical feature extraction and explore the association between the hierarchical labels and the text. In this paper, we propose a Label-based Attention for Hierarchical Mutlti-label Text Classification Neural Network (LA-HCN), where the novel label-based attention module is designed to hierarchically extract important information from the text based on the labels from different hierarchy levels. Besides, hierarchical information is shared across levels while preserving the hierarchical label-based information. Separate local and global document embeddings are obtained and used to facilitate the respective local and global classifications. In our experiments, LA-HCN outperforms other state-of-the-art neural network-based HMTC algorithms on four public HMTC datasets. The ablation study also demonstrates the effectiveness of the proposed label-based attention module as well as the novel local and global embeddings and classifications. By visualizing the learned attention (words), we find that LA-HCN is able to extract meaningful information corresponding to the different labels which provides explainability that may be helpful for the human analyst.

Motivation & Objective

• To address the limitation of existing HMTC models that focus only on classifier design while neglecting hierarchical feature extraction.
• To improve model interpretability and performance by learning label-specific attention mechanisms at each level of the label hierarchy.
• To reduce error propagation in hierarchical classification by using separate local and global document embeddings.
• To enable effective sharing of hierarchical label information across levels while preserving label-specific feature representations.
• To demonstrate that label-based attention leads to more meaningful and explainable feature extraction compared to standard attention mechanisms.

Proposed method

• Introduces a label-based attention mechanism that dynamically focuses on words relevant to specific labels at each hierarchical level.
• Employs component embeddings as intermediate representations to bridge word-level features and hierarchical labels, enhancing label-specific feature learning.
• Generates separate local and global document embeddings to support distinct local and global classification heads, reducing error propagation.
• Shares hierarchical label information across levels while maintaining label-specific feature representations through the component mechanism.
• Uses a dual-head classification strategy—local classification per level and global classification over the entire hierarchy—fused via early concatenation of embeddings.
• Applies a hierarchical attention mechanism that conditions word attention on the label being predicted at each level, enabling interpretable attention maps.

Experimental results

Research questions

• RQ1 Can label-based attention mechanisms improve feature extraction in hierarchical multi-label text classification by focusing on label-specific content?
• RQ2 Does separating local and global document embeddings reduce error propagation in hierarchical classification?
• RQ3 How effective is the proposed component embedding mechanism in capturing label dependencies across hierarchical levels?
• RQ4 Can the model achieve better performance and interpretability compared to state-of-the-art HMTC models?
• RQ5 To what extent does hierarchical information sharing enhance classification accuracy while preserving label-specific relevance?

Key findings

• LA-HCN outperforms all state-of-the-art neural network-based HMTC models on four public benchmark datasets, achieving the highest AU-PRC scores.
• The ablation study confirms that the component embedding mechanism significantly improves performance, with LA-HCN NC (no components) performing the worst among variants.
• Global classification consistently outperforms local classification in three out of four datasets, and combining both yields the best results.
• Visualization of attention shows that LA-HCN correctly highlights words relevant to specific labels—e.g., 'yoga' for 'Exercise' and 'spiritual' for 'Philosophy'—demonstrating interpretability.
• Unlike HARNN and HAN, which apply uniform attention across labels, LA-HCN learns distinct attention patterns per label, preserving label-specific relevance across levels.
• The model effectively maintains important features like 'yoga' at deeper levels even when they are not relevant at broader levels, showing robust hierarchical feature propagation.