[Paper Review] Bridging Ordinary-Label Learning and Complementary-Label Learning

Yasuhiro Katsura, Masato Uchida|arXiv (Cornell University)|Feb 6, 2020

Text and Document Classification Technologies21 references4 citations

TL;DR

This paper unifies ordinary-label and complementary-label learning by leveraging duality and additivity in loss functions for one-versus-all and pairwise classification. It derives a new classification risk and error bound that monotonically decreases with more complementary labels, demonstrating that learning from complementary labels can match or exceed performance from ordinary labels under the same conditions.

ABSTRACT

A supervised learning framework has been proposed for the situation where each training data is provided with a complementary label that represents a class to which the pattern does not belong. In the existing literature, complementary-label learning has been studied independently from ordinary-label learning, which assumes that each training data is provided with a label representing the class to which the pattern belongs. However, providing a complementary label should be treated as equivalent to providing the rest of all the labels as the candidates of the one true class. In this paper, we focus on the fact that the loss functions for one-versus-all and pairwise classification corresponding to ordinary-label learning and complementary-label learning satisfy certain additivity and duality, and provide a framework which directly bridge those existing supervised learning frameworks. Further, we derive classification risk and error bound for any loss functions which satisfy additivity and duality.

Motivation & Objective

• To unify ordinary-label and complementary-label learning under a single theoretical framework.
• To analyze the classification risk and error bound for loss functions satisfying additivity and duality.
• To generalize existing frameworks that treat ordinary and complementary labeling as separate problems.
• To demonstrate that increasing the number of complementary labels improves classification performance.
• To provide a theoretical foundation for privacy-preserving learning using multiple complementary labels.

Proposed method

• Proposes a unified loss function framework based on duality and additivity between one-versus-all and pairwise classification.
• Defines a data-generation probability model that assigns multiple complementary labels uniformly across classes.
• Derives a classification risk and error bound using origin-symmetric sigmoid loss functions.
• Uses surrogate classifiers to estimate empirical risk and test error in experiments.
• Applies theoretical bounds with δ = 0.1 and Lℓ = 1.0, assuming sufficient model capacity.
• Employs MLP and DenseNet on MNIST, Fashion-MNIST, Kuzushiji-MNIST, and CIFAR-10 with standard hyperparameters.

Experimental results

Research questions

• RQ1 How can ordinary-label and complementary-label learning be unified under a single theoretical framework?
• RQ2 What properties of loss functions enable a direct bridge between one-versus-all and pairwise classification?
• RQ3 How does the number of complementary labels affect classification risk and error?
• RQ4 Can the derived error bound accurately reflect empirical performance trends?
• RQ5 What is the impact of label generation uniformity on the generalization of complementary-label learning?

Key findings

• The classification error monotonically decreases as the number of complementary labels increases, indicating improved performance with more labels.
• The theoretical error bound for one-versus-all and pairwise classification is similarly tight, validating the robustness of the derived bound.
• Experimental results show a consistent trend where accuracy improves with more complementary labels, supporting the theoretical findings.
• The experimental error and theoretical bound exhibit similar qualitative shapes, confirming the bound's relevance to real-world performance.
• The derived classification risk is simpler and more general than prior formulations in Ishida et al. (2017, 2019), Cao and Xu (2020), and Feng et al. (2020).
• The framework supports privacy-preserving learning by preventing label distribution leakage through uniform, multiple complementary label assignment.