[Paper Review] Regularizing Model Complexity and Label Structure for Multi-Label Text Classification

Bingyu Wang, Cheng Li+2 more|arXiv (Cornell University)|May 1, 2017

Text and Document Classification Technologies4 references3 citations

TL;DR

This paper proposes a two-stage framework for multi-label text classification that combines model regularization during training and label search space regularization during prediction. By applying Elastic-net and early stopping to reduce overfitting, and using support inference with F-optimizer GFM to improve F1 performance, the method achieves state-of-the-art results across multiple benchmark datasets, demonstrating that label structure regularization is more impactful than complex label dependency modeling in practice.

ABSTRACT

Multi-label text classification is a popular machine learning task where each document is assigned with multiple relevant labels. This task is challenging due to high dimensional features and correlated labels. Multi-label text classifiers need to be carefully regularized to prevent the severe over-fitting in the high dimensional space, and also need to take into account label dependencies in order to make accurate predictions under uncertainty. We demonstrate significant and practical improvement by carefully regularizing the model complexity during training phase, and also regularizing the label search space during prediction phase. Specifically, we regularize the classifier training using Elastic-net (L1+L2) penalty for reducing model complexity/size, and employ early stopping to prevent overfitting. At prediction time, we apply support inference to restrict the search space to label sets encountered in the training set, and F-optimizer GFM to make optimal predictions for the F1 metric. We show that although support inference only provides density estimations on existing label combinations, when combined with GFM predictor, the algorithm can output unseen label combinations. Taken collectively, our experiments show state of the art results on many benchmark datasets. Beyond performance and practical contributions, we make some interesting observations. Contrary to the prior belief, which deems support inference as purely an approximate inference procedure, we show that support inference acts as a strong regularizer on the label prediction structure. It allows the classifier to take into account label dependencies during prediction even if the classifiers had not modeled any label dependencies during training.

Motivation & Objective

• To address the challenges of high-dimensional, sparse text features and correlated labels in multi-label text classification.
• To improve F1-score optimization in multi-label prediction, especially under label imbalance and partial credit for near-correct predictions.
• To reduce model complexity and overfitting in high-capacity classifiers without relying solely on complex label dependency modeling.
• To investigate whether support inference acts as a strong regularizer beyond being an approximate inference method.
• To demonstrate that combining training regularization with prediction-phase label space regularization yields superior performance

Proposed method

• Applies Elastic-net (L1 + L2) regularization during training to reduce model complexity and prevent overfitting in high-dimensional feature spaces.
• Employs early stopping as a form of implicit regularization to halt training before overfitting occurs.
• Uses support inference at prediction time to restrict the label search space to only those label combinations observed in the training set, effectively regularizing the prediction structure.
• Integrates the F-optimizer GFM (Greedy F-Measure) predictor to compute optimal label sets that maximize the F1 metric.
• Combines support inference with GFM to enable predictions of unseen label combinations while maintaining F1-optimality.
• Uses joint probability estimation from the classifier to sample label sets for GFM, ensuring F1-optimality even when the model does not explicitly model label dependencies.

Experimental results

Research questions

• RQ1 Does support inference act as a strong regularizer on label prediction structure, even when the classifier does not model label dependencies during training?
• RQ2 Can combining training-phase regularization (e.g., Elastic-net, early stopping) with prediction-phase regularization (support inference + GFM) outperform models that rely solely on complex label dependency structures?
• RQ3 How effective is the GFM predictor when used in conjunction with support inference, especially for models like Binary Relevance (BR) and Probabilistic Classifier Chains (PCC)?
• RQ4 To what extent does the performance of state-of-the-art models like CRF or CBM depend on label pair interactions versus support inference?
• RQ5 Does the proposed framework generalize across diverse multi-label datasets, including medical, news, and patent text collections?

Key findings

• Support inference acts as a powerful regularizer on label prediction structure, enabling effective modeling of label dependencies during prediction even when the classifier was trained without explicit dependency modeling.
• The combination of support inference and GFM consistently improves F1 performance across all tested classifiers (BR, PCC, CBM, CRF) and datasets, outperforming methods that rely only on internal label dependency structures.
• Removing label pair interactions from CRF results in minimal performance drop, indicating that support inference contributes significantly to CRF’s success, not just the pairwise terms.
• The method achieves state-of-the-art F1 scores on benchmark datasets including OHSUMED (49.5), WIPO (74.3), ENRON (62.5), and MEDICAL (82.6), with CBM achieving 82.6 F1 on MEDICAL.
• Model size is significantly reduced through Elastic-net and early stopping, making high-capacity models like CBM and CRF more practical despite high-dimensional features.
• Even without fine-tuning, the SPEN neural network model underperforms compared to the proposed framework, suggesting that overfitting is a major issue in high-capacity models without proper regularization.