[Paper Review] Large-scale Multi-label Learning with Missing Labels

Hsiang‐Fu Yu, Prateek Jain+2 more|arXiv (Cornell University)|Jul 18, 2013

Text and Document Classification Technologies23 references369 citations

TL;DR

This paper proposes a scalable empirical risk minimization framework for large-scale multi-label learning with missing labels, using low-rank matrix modeling and trace-norm regularization. It achieves state-of-the-art performance on benchmark datasets like Wikipedia, with efficient optimization via conjugate gradient and alternating minimization, and provides tight theoretical excess risk bounds under random label missingness.

ABSTRACT

The multi-label classification problem has generated significant interest in recent years. However, existing approaches do not adequately address two key challenges: (a) the ability to tackle problems with a large number (say millions) of labels, and (b) the ability to handle data with missing labels. In this paper, we directly address both these problems by studying the multi-label problem in a generic empirical risk minimization (ERM) framework. Our framework, despite being simple, is surprisingly able to encompass several recent label-compression based methods which can be derived as special cases of our method. To optimize the ERM problem, we develop techniques that exploit the structure of specific loss functions - such as the squared loss function - to offer efficient algorithms. We further show that our learning framework admits formal excess risk bounds even in the presence of missing labels. Our risk bounds are tight and demonstrate better generalization performance for low-rank promoting trace-norm regularization when compared to (rank insensitive) Frobenius norm regularization. Finally, we present extensive empirical results on a variety of benchmark datasets and show that our methods perform significantly better than existing label compression based methods and can scale up to very large datasets such as the Wikipedia dataset.

Motivation & Objective

• To address the dual challenges of large-scale label spaces (up to millions of labels) and missing labels in multi-label learning.
• To develop a unified, flexible framework that subsumes existing label-compression methods as special cases.
• To design efficient optimization algorithms that scale to massive datasets like Wikipedia.
• To provide formal generalization guarantees (excess risk bounds) even when labels are partially missing.
• To empirically demonstrate superior performance over existing label-compression and multi-label methods on diverse benchmark datasets.

Proposed method

• Formulates multi-label learning as an empirical risk minimization (ERM) problem with a low-rank linear model $ Z \in \mathbb{R}^{d \times L} $, where predictions are $ \mathbf{y}^{\text{pred}} = Z^T \mathbf{x} $.
• Uses trace-norm regularization to promote low-rank solutions and improve generalization, especially under label sparsity.
• Employs alternating minimization and conjugate gradient methods to optimize the non-convex ERM problem with structured loss functions.
• Derives a closed-form solution for the squared $ L_2 $ loss case, showing equivalence to the CPLST method of Chen & Lin (2012) as a special case.
• Extends the framework to handle missing labels by assuming uniform random observation of labels, enabling theoretical analysis via random matrix theory.
• Designs a scalable algorithm that is $ O(\bar{d}) $ faster than direct computation, where $ \bar{d} $ is the average number of non-zero features per instance.

Experimental results

Research questions

• RQ1 Can a unified ERM framework effectively handle both massive label spaces and missing labels in multi-label learning?
• RQ2 How does trace-norm regularization compare to Frobenius norm regularization in terms of generalization under label sparsity?
• RQ3 Can the proposed framework achieve state-of-the-art performance on large-scale datasets like Wikipedia with missing labels?
• RQ4 What is the theoretical excess risk bound of the trace-norm regularized ERM formulation under random label missingness?
• RQ5 How does the efficiency of the optimization algorithm scale with data size and sparsity?

Key findings

• The proposed method achieves significantly better performance than existing label-compression methods on benchmark datasets, including the Wikipedia dataset with over 100k labels.
• On the bibtex dataset with 50% missing labels, the method achieves an average AUC of 0.8724 using the squared hinge loss, outperforming baseline methods.
• For the autofood dataset with 40% label sparsity, the method achieves an average AUC of 0.9260 under the logistic loss, surpassing all baselines.
• Theoretical analysis shows that trace-norm regularization leads to tighter excess risk bounds than Frobenius norm regularization for isotropic data distributions.
• The optimization algorithm is $ O(\bar{d}) $ faster than direct computation, enabling efficient scaling to large, sparse datasets.
• The framework generalizes existing label-compression methods, such as CPLST, as special cases under the squared $ L_2 $ loss.