[Paper Review] HPT: Hierarchy-aware Prompt Tuning for Hierarchical Text Classification

Zihan Wang, Peiyi Wang+5 more|arXiv (Cornell University)|Apr 28, 2022

Text and Document Classification Technologies4 citations

TL;DR

This paper proposes HPT, a hierarchy-aware prompt tuning method for hierarchical text classification that integrates label hierarchy knowledge into soft prompts and uses a zero-bounded multi-label cross entropy loss to align with masked language modeling objectives. HPT achieves state-of-the-art performance on three benchmark datasets and significantly improves performance under label imbalance and low-resource settings.

ABSTRACT

Hierarchical text classification (HTC) is a challenging subtask of multi-label classification due to its complex label hierarchy. Recently, the pretrained language models (PLM)have been widely adopted in HTC through a fine-tuning paradigm. However, in this paradigm, there exists a huge gap between the classification tasks with sophisticated label hierarchy and the masked language model (MLM) pretraining tasks of PLMs and thus the potentials of PLMs can not be fully tapped. To bridge the gap, in this paper, we propose HPT, a Hierarchy-aware Prompt Tuning method to handle HTC from a multi-label MLM perspective. Specifically, we construct a dynamic virtual template and label words that take the form of soft prompts to fuse the label hierarchy knowledge and introduce a zero-bounded multi-label cross entropy loss to harmonize the objectives of HTC and MLM. Extensive experiments show HPT achieves state-of-the-art performances on 3 popular HTC datasets and is adept at handling the imbalance and low resource situations. Our code is available at https://github.com/wzh9969/HPT.

Motivation & Objective

• To address the gap between masked language modeling pretraining and hierarchical text classification fine-tuning in pretrained language models (PLMs).
• To bridge the hierarchy and flat gap by incorporating label hierarchy structure into soft prompts using depth and width information.
• To resolve the multi-label and multi-class gap by reformulating HTC as a multi-label masked language modeling task.
• To improve performance on imbalanced and low-resource hierarchical text classification scenarios.

Proposed method

• Constructs dynamic virtual templates and label words as continuous soft prompts that encode label hierarchy depth and width information.
• Integrates label hierarchy knowledge into the prompt design by embedding structural features (depth, width) into the soft prompt tokens.
• Employs a zero-bounded multi-label cross entropy loss to simultaneously increase scores for correct labels and decrease scores for incorrect labels.
• Transforms hierarchical text classification into a hierarchy-aware multi-label masked language modeling task to better align with PLM pretraining objectives.
• Uses pretrained language models as the backbone, with prompt tuning instead of full fine-tuning to preserve pretraining knowledge.
• Applies the prompt-tuned model to downstream HTC tasks by masking the label prediction head and optimizing the soft prompt parameters.

Experimental results

Research questions

• RQ1 Can prompt tuning be effectively adapted to hierarchical text classification by incorporating label hierarchy structure?
• RQ2 How does a hierarchy-aware prompt design improve performance on imbalanced label hierarchies?
• RQ3 Can a zero-bounded multi-label cross entropy loss better align the objectives of HTC and masked language modeling?
• RQ4 Does HPT outperform existing fine-tuning and prompt-based methods under low-resource and long-tail label scenarios?
• RQ5 To what extent does HPT mitigate the pretraining-fine-tuning gap in hierarchical text classification?

Key findings

• HPT achieves state-of-the-art performance on three popular hierarchical text classification datasets: NYT, RCV1-V2, and DBPedia.
• On the NYT dataset, HPT improves macro-F1 scores for medium-depth label clusters (depth 3 and 4), which are typically underperforming in baseline models.
• HPT significantly reduces the performance gap for low-resource labels, showing strong robustness in long-tail scenarios.
• In low-resource settings with only 10% of training data, HPT outperforms all baselines and maintains lower standard deviation, indicating better stability.
• On RCV1-V2, HPT achieves a 6.09-point higher macro-F1 score than the next-best baseline in low-resource settings, compared to a 2.13-point gap in full-resource settings.