[Paper Review] Does Head Label Help for Long-Tailed Multi-Label Text Classification

Lin Xiao, Xiangliang Zhang+3 more|arXiv (Cornell University)|Jan 24, 2021

Text and Document Classification Technologies33 references4 citations

TL;DR

This paper proposes a Head-to-Tail Network (HTTN) that transfers meta-knowledge from data-rich head labels to data-poor tail labels in long-tailed multi-label text classification. By learning a parameter transfer mapping from few-shot to many-shot label classifiers, HTTN improves tail label performance without retraining, achieving state-of-the-art results on three benchmark datasets, especially on extreme long-tail labels with only 2–6 training instances.

ABSTRACT

Multi-label text classification (MLTC) aims to annotate documents with the most relevant labels from a number of candidate labels. In real applications, the distribution of label frequency often exhibits a long tail, i.e., a few labels are associated with a large number of documents (a.k.a. head labels), while a large fraction of labels are associated with a small number of documents (a.k.a. tail labels). To address the challenge of insufficient training data on tail label classification, we propose a Head-to-Tail Network (HTTN) to transfer the meta-knowledge from the data-rich head labels to data-poor tail labels. The meta-knowledge is the mapping from few-shot network parameters to many-shot network parameters, which aims to promote the generalizability of tail classifiers. Extensive experimental results on three benchmark datasets demonstrate that HTTN consistently outperforms the state-of-the-art methods. The code and hyper-parameter settings are released for reproducibility

Motivation & Objective

• To address the challenge of poor generalization in long-tailed multi-label text classification, where tail labels suffer from insufficient training data.
• To investigate whether head labels can effectively help in learning robust classifiers for tail labels despite data imbalance.
• To develop a method that leverages label dependency and meta-knowledge transfer from head to tail labels without retraining the full model for new labels.
• To improve tail label performance while maintaining strong head label performance, avoiding dominance by high-frequency labels.

Proposed method

• HTTN learns meta-knowledge as a mapping from classifier weights trained on few-shot head labels to those trained on many-shot head labels.
• This meta-knowledge, denoted as $W_{\text{transfer}}$, is distilled from multiple sampled prototypes of head labels to generalize across label categories.
• The model transfers this knowledge to tail labels by initializing their classifiers using $W_{\text{transfer}}$ and fine-tuning on their limited data.
• An attention module is used to model label correlations between head and tail labels, enhancing feature representation and prediction accuracy.
• An ensemble mechanism is applied to multiple HTTN instances with different sampled head prototypes to improve robustness and diversity.
• The method requires only training on head labels; new tail labels can be classified immediately using the pre-learned meta-knowledge.

Experimental results

Research questions

• RQ1 Can head labels provide useful meta-knowledge that improves classification performance for low-resource tail labels in long-tailed multi-label text classification?
• RQ2 How does label dependency between head and tail labels influence the transfer of knowledge in multi-label learning?
• RQ3 Does transferring a generic, class-irrelevant mapping from few-shot to many-shot learning improve generalization on tail labels?
• RQ4 Can the proposed method achieve state-of-the-art performance without retraining the entire model when new tail labels are introduced?
• RQ5 How does the number of sampled head prototypes affect the robustness and generalization of the transfer knowledge?

Key findings

• HTTN consistently outperforms seven state-of-the-art baselines across three benchmark datasets (AAPD, RCV1, and another), especially on tail labels.
• On extreme tail labels with only 2–6 training instances, HTTN achieves non-zero positive predictions, while Joint and BBN models predict zero for all such labels.
• The ensemble version of HTTN (EHTTN) achieves the best performance, demonstrating that diversity and robustness are enhanced through multiple model instances.
• Increasing the number of sampled prototypes ($S$) from 5 to 20 significantly improves F1-score, indicating that richer meta-knowledge leads to better generalization.
• The attention module and fine-tuning components are both essential: removing them degrades performance, showing their role in capturing label correlations and improving accuracy.
• When $l_{\text{tail}}$ is smaller (i.e., fewer tail labels), HTTN performs better, as more head labels are available to distill richer meta-knowledge.