[Paper Review] Unbiased Loss Functions for Extreme Classification With Missing Labels

Erik Schultheis, Mohammadreza Qaraei+2 more|arXiv (Cornell University)|Jul 1, 2020

Text and Document Classification Technologies37 references4 citations

TL;DR

This paper proposes unbiased loss functions for extreme multi-label classification (XMC) that correct for missing labels and long-tail label imbalance. By deriving label-specific weighting factors for common loss functions—such as squared hinge and binary cross-entropy—it enables more accurate tail-label prediction while maintaining performance on standard metrics, achieving up to 20% improvement on benchmark datasets.

ABSTRACT

The goal in extreme multi-label classification (XMC) is to tag an instance with a small subset of relevant labels from an extremely large set of possible labels. In addition to the computational burden arising from large number of training instances, features and labels, problems in XMC are faced with two statistical challenges, (i) large number of 'tail-labels' -- those which occur very infrequently, and (ii) missing labels as it is virtually impossible to manually assign every relevant label to an instance. In this work, we derive an unbiased estimator for general formulation of loss functions which decompose over labels, and then infer the forms for commonly used loss functions such as hinge- and squared-hinge-loss and binary cross-entropy loss. We show that the derived unbiased estimators, in the form of appropriate weighting factors, can be easily incorporated in state-of-the-art algorithms for extreme classification, thereby scaling to datasets with hundreds of thousand labels. However, empirically, we find a slightly altered version that gives more relative weight to tail labels to perform even better. We suspect is due to the label imbalance in the dataset, which is not explicitly addressed by our theoretically derived estimator. Minimizing the proposed loss functions leads to significant improvement over existing methods (up to 20% in some cases) on benchmark datasets in XMC.

Motivation & Objective

• Address the statistical challenges in extreme multi-label classification (XMC), including long-tail label distribution and missing labels in training data.
• Derive theoretically unbiased estimators for loss functions that decompose over individual labels, such as hinge, squared-hinge, and binary cross-entropy loss.
• Develop a practical method that incorporates these unbiased estimators into state-of-the-art XMC algorithms without sacrificing computational efficiency.
• Improve performance on tail labels—commonly under-predicted due to data scarcity and label missingness—while maintaining competitive performance on standard metrics.

Proposed method

• Derive an unbiased estimator for general label-decomposable loss functions by modeling the missing label mechanism via a propensity score, assuming missingness is independent of the true label given the observed features.
• Transform the original loss functions into weighted forms using inverse propensity scores to correct for missing label bias, ensuring the expected loss over the observed data matches the true expected loss.
• Apply the derived weighting scheme to squared-hinge and binary cross-entropy losses, which are commonly used in linear SVM and logistic regression-based XMC models.
• Introduce a modified weighting scheme that increases relative weight to tail labels, empirically improving performance despite the theoretical estimator not explicitly modeling label imbalance.
• Integrate the weighted loss into a one-vs-rest linear SVM framework, enabling efficient training with second-order optimization.
• Use l2 regularization to stabilize training and improve generalization, especially in high-dimensional, sparse settings common in XMC.

Experimental results

Research questions

• RQ1 Can we derive an unbiased estimator for commonly used loss functions in extreme multi-label classification when labels are missing at random?
• RQ2 How do the theoretically derived unbiased loss functions perform in practice when applied to real-world XMC datasets with long-tailed label distributions?
• RQ3 Does incorporating label-specific weighting factors derived from propensity models improve prediction performance on tail labels compared to standard baselines?
• RQ4 Can the proposed method maintain or improve performance on standard metrics like precision and nDCG while significantly enhancing tail-label detection?
• RQ5 Why does a slightly modified weighting scheme—giving more weight to tail labels—outperform the theoretically unbiased estimator in practice?

Key findings

• The proposed method achieves up to 20% relative improvement in performance on benchmark XMC datasets, particularly on tail labels, compared to existing state-of-the-art methods.
• Incorporating the unbiased loss with label-specific weights into a one-vs-rest linear SVM framework significantly improves tail-label prediction without degrading performance on head labels.
• The method outperforms ProXML, a specialized algorithm for tail-label detection, while being nearly two orders of magnitude faster due to compatibility with second-order optimization.
• Empirical results show that a modified weighting scheme—increasing relative weight to tail labels—yields better performance than the theoretically unbiased estimator, likely due to unaddressed label imbalance in the theoretical derivation.
• The method maintains competitive performance on standard metrics such as precision and nDCG, demonstrating that improvements on tail labels do not come at the cost of overall accuracy.
• The derived unbiased estimators are applicable to a wide range of loss functions that decompose over labels, including hinge, squared-hinge, and binary cross-entropy, enabling broad applicability in XMC.