[Paper Review] Provably Consistent Partial-Label Learning

Lei Feng, Jiaqi Lv+6 more|arXiv (Cornell University)|Jul 17, 2020

Text and Document Classification Technologies78 references53 citations

TL;DR

The paper introduces a generation model for partial-label data and two provably consistent PLL methods (risk-consistent and classifier-consistent) that work with deep networks, plus estimation error bounds and empirical validation.

ABSTRACT

Partial-label learning (PLL) is a multi-class classification problem, where each training example is associated with a set of candidate labels. Even though many practical PLL methods have been proposed in the last two decades, there lacks a theoretical understanding of the consistency of those methods-none of the PLL methods hitherto possesses a generation process of candidate label sets, and then it is still unclear why such a method works on a specific dataset and when it may fail given a different dataset. In this paper, we propose the first generation model of candidate label sets, and develop two novel PLL methods that are guaranteed to be provably consistent, i.e., one is risk-consistent and the other is classifier-consistent. Our methods are advantageous, since they are compatible with any deep network or stochastic optimizer. Furthermore, thanks to the generation model, we would be able to answer the two questions above by testing if the generation model matches given candidate label sets. Experiments on benchmark and real-world datasets validate the effectiveness of the proposed generation model and two PLL methods.

Motivation & Objective

• Motivate PLL by practical difficulty of obtaining exact labels and the need for theoretical guarantees.
• Propose a data-generation model for partially labeled data that ensures the correct label is within the candidate set.
• Develop two PLL methods with formal consistency guarantees (risk-consistent and classifier-consistent).
• Provide risk estimation bounds and demonstrate convergence to the true risk minimizer.
• Empirically validate the methods on benchmark and real-world partially labeled datasets.

Proposed method

• Define a probabilistic model for generating candidate label sets that always include the true label (Eq. 5).
• Derive a risk-consistent PLL method using importance reweighting (Eq. 8–9) and implement with cross-entropy loss (Eq. 9).
• Derive a classifier-consistent PLL method using a transition matrix Q and a risk estimator (Eq. 11–12).
• Provide estimation error bounds for both RC (Theorem 4) and CC (Theorem 6).
• Show RC tends to outperform CC with deep models due to tighter bounds and confidence estimation.
• Ensure methods are model- and optimizer-agnostic for compatibility with modern networks.

Experimental results

Research questions

• RQ1 Can a explicit data-generation process for partially labeled data be defined to guarantee PLL assumptions hold?
• RQ2 Can we design PLL methods that are provably risk-consistent and classifier-consistent under this generation model?
• RQ3 What are the estimation error bounds and convergence properties of the proposed methods?
• RQ4 How do the proposed methods perform empirically on benchmarks and real-world PLL datasets, and how does model complexity affect performance?

Key findings

• Two provably consistent PLL methods are proposed: a risk-consistent method (RC) and a classifier-consistent method (CC).
• RC provides a tighter estimation error bound than CC and performs better with deep networks in practice.
• An explicit data-generation model for partial labels is shown to satisfy the PLL core assumption that the correct label lies in the candidate set (Theorems 1–2).
• RC uses importance reweighting with p(y=i|x) approximated by softmax outputs and zeroing non-candidate labels (Eq. 10).
• CC uses a transition matrix Q to relate ordinary labels to candidate-labels and yields an unbiased classifier-consistent estimator (Theorems 5–6).
• Experiments on MNIST, Kuzushiji-MNIST, Fashion-MNIST, CIFAR-10, and real-world PLL datasets validate effectiveness across neural and linear models.