[Paper Review] Learning from Label Proportions with Generative Adversarial Networks

Jiabin Liu, Bo Wang+3 more|arXiv (Cornell University)|Sep 5, 2019

Model Reduction and Neural Networks6 citations

TL;DR

This paper proposes LLP-GAN, a generative adversarial network framework for learning from label proportions (LLP) without restrictive distributional assumptions. By leveraging adversarial training with a discriminator that classifies both real instances and generated fake samples, LLP-GAN end-to-end learns a robust instance-level classifier, achieving state-of-the-art performance on benchmark datasets with low computational complexity and global optimality under mild assumptions.

ABSTRACT

In this paper, we leverage generative adversarial networks (GANs) to derive an effective algorithm LLP-GAN for learning from label proportions (LLP), where only the bag-level proportional information in labels is available. Endowed with end-to-end structure, LLP-GAN performs approximation in the light of an adversarial learning mechanism, without imposing restricted assumptions on distribution. Accordingly, we can directly induce the final instance-level classifier upon the discriminator. Under mild assumptions, we give the explicit generative representation and prove the global optimality for LLP-GAN. Additionally, compared with existing methods, our work empowers LLP solver with capable scalability inheriting from deep models. Several experiments on benchmark datasets demonstrate vivid advantages of the proposed approach.

Motivation & Objective

• To address the challenge of learning instance-level classifiers from only bag-level label proportions, a weakly supervised setting where full instance labels are unavailable.
• To overcome limitations of prior LLP methods, such as strict distributional assumptions and scalability issues from NP-hard optimization.
• To leverage the representational power of deep networks and adversarial training to improve generalization and robustness in LLP settings.
• To establish theoretical guarantees for the proposed method, including global optimality and explicit generative representation under mild assumptions.
• To demonstrate scalability and performance gains over existing state-of-the-art methods on large-scale benchmark datasets.

Proposed method

• LLP-GAN employs a GAN framework where the generator synthesizes fake samples to train the discriminator to distinguish real instances from generated ones.
• The discriminator is trained to classify real instances into K classes and detect fake samples as a (K+1)th class, enabling end-to-end learning of the final classifier.
• A lower bound on the discriminator's loss is derived, linking prior class proportions to posterior class likelihoods through a decomposition representation.
• The generator learns the underlying data distribution via adversarial training without assuming i.i.d. bags, enhancing flexibility and generalization.
• The framework incorporates a 11-way softmax over-parameterized classifier head and uses global average pooling and 1×1 convolutions for feature refinement.
• The method avoids explicit variational inference or likelihood maximization, instead relying on adversarial equilibrium between generator and discriminator.

Experimental results

Research questions

• RQ1 Can a GAN-based framework effectively learn instance-level classifiers from only label proportions without strong distributional assumptions?
• RQ2 How does the proposed adversarial training mechanism in LLP-GAN ensure global optimality and stable convergence in the LLP setting?
• RQ3 What is the relationship between prior label proportions and posterior class likelihoods in the discriminator's output?
• RQ4 How does LLP-GAN compare in performance and scalability to existing deep and shallow LLP methods on large-scale datasets?
• RQ5 To what extent does the proposed method remain robust under random bag assignments and varying bag sizes?

Key findings

• LLP-GAN achieves state-of-the-art performance on MNIST, CIFAR-10, SVHN, and CIFAR-100, with test error rates as low as 0.047% on MNIST under binary classification.
• On CIFAR-10, LLP-GAN reduces test error from 22.59% (DLLP) to 1.61% under larger bag sizes, demonstrating superior scalability and robustness.
• The method maintains high stability across multiple random bag assignments, with accuracy on MNIST varying by only 0.4% (from 98.94% to 96.65%) under different randomizations.
• LLP-GAN achieves global optimality under mild assumptions, with a theoretically grounded decomposition of class likelihoods in terms of prior proportions.
• The addition of entropy regularization to DLLP was found redundant, as DLLP already achieved low instance-level entropy, suggesting inherent stability in the baseline.
• The framework shows low computational complexity and strong scalability, outperforming SVM-based methods like InvCal and alter-∝ SVM, which suffer from NP-hard optimization.