[Paper Review] Enhancing Label Correlation Feedback in Multi-Label Text Classification via Multi-Task Learning

Ximing Zhang, Qianwen Zhang+3 more|arXiv (Cornell University)|Jun 6, 2021

Text and Document Classification Technologies19 references4 citations

TL;DR

This paper proposes LACO, a multi-task learning framework for multi-label text classification that enhances label correlation feedback via joint document-label embedding and two auxiliary tasks: Pairwise Label Co-occurrence Prediction (PLCP) and Conditional Label Co-occurrence Prediction (CLCP). The method outperforms strong baselines on AAPD and RCV1-V2, showing improved generalization, especially for low-frequency labels and unseen label combinations, with faster convergence and better label diversity.

ABSTRACT

In multi-label text classification (MLTC), each given document is associated with a set of correlated labels. To capture label correlations, previous classifier-chain and sequence-to-sequence models transform MLTC to a sequence prediction task. However, they tend to suffer from label order dependency, label combination over-fitting and error propagation problems. To address these problems, we introduce a novel approach with multi-task learning to enhance label correlation feedback. We first utilize a joint embedding (JE) mechanism to obtain the text and label representation simultaneously. In MLTC task, a document-label cross attention (CA) mechanism is adopted to generate a more discriminative document representation. Furthermore, we propose two auxiliary label co-occurrence prediction tasks to enhance label correlation learning: 1) Pairwise Label Co-occurrence Prediction (PLCP), and 2) Conditional Label Co-occurrence Prediction (CLCP). Experimental results on AAPD and RCV1-V2 datasets show that our method outperforms competitive baselines by a large margin. We analyze low-frequency label performance, label dependency, label combination diversity and coverage speed to show the effectiveness of our proposed method on label correlation learning.

Motivation & Objective

• To address the limitations of sequence-to-sequence models in multi-label text classification, such as label order dependency, overfitting to frequent label combinations, and error propagation.
• To improve generalization for low-frequency labels by explicitly modeling label correlations without relying on sequential generation.
• To enhance label correlation feedback through auxiliary tasks that capture both second-order and high-order label co-occurrence patterns.
• To develop a model that learns discriminative document and label representations jointly while maintaining order-independence of labels.

Proposed method

• Utilizes a shared transformer-based encoder with a joint embedding (JE) mechanism to jointly learn document and label representations in a shared space.
• Employs a document-label cross attention (CA) mechanism to generate label-specific document representations that retain discriminative features.
• Introduces Pairwise Label Co-occurrence Prediction (PLCP) to model second-order label correlations by predicting whether two labels co-occur in a given sample.
• Proposes Conditional Label Co-occurrence Prediction (CLCP) to model high-order label correlations by predicting the relevance of unknown labels given a partial set of relevant labels.
• Trains the multi-task learning framework end-to-end, jointly optimizing the main multi-label classification task with the two auxiliary correlation learning tasks.
• Uses BERT-based encoders and trains the model with cross-entropy loss on the main task and auxiliary tasks, enabling feature sharing and improved representation learning.

Experimental results

Research questions

• RQ1 Can a multi-task learning framework effectively enhance label correlation feedback in multi-label text classification without relying on sequence generation?
• RQ2 To what extent do auxiliary tasks like PLCP and CLCP improve generalization, especially for low-frequency and unseen label combinations?
• RQ3 How does the proposed method compare to sequence-to-sequence models in terms of label diversity, convergence speed, and robustness to label order?
• RQ4 Does joint document-label embedding with cross-attention lead to better discriminative representations than independent label modeling?

Key findings

• LACO outperforms competitive baselines on both AAPD and RCV1-V2 datasets, achieving higher Micro-F1 scores and better subset accuracy.
• The model with both PLCP and CLCP auxiliary tasks achieves the highest subset accuracy (0.644 on RCV1-V2) and generates more diverse label combinations (241 unique combinations on RCV1-V2) than Seq2Seq-based models.
• LACO produces significantly more diverse label combinations (321 on RCV1-V2) compared to Seq2Seq T (87), indicating better generalization to unseen label combinations.
• The KL divergence between the model’s predicted label distribution and the ground truth is lower for LACO (0.76 on RCV1-V2 test set) than for Seq2Seq T (0.94), showing better fitting of label dependency patterns.
• LACO converges faster than other BERT-based models, as shown in the convergence speed curve, with multi-task learning accelerating training.
• The model shows strong performance on low-frequency labels, with improved prediction capability due to enhanced label correlation feedback from auxiliary tasks.