[Paper Review] A Variational Approach for Learning from Positive and Unlabeled Data

Hui Chen, Fangqing Liu+3 more|arXiv (Cornell University)|Jun 3, 2019

Machine Learning and Data Classification28 references22 citations

TL;DR

This paper proposes a variational approach for positive-unlabeled (PU) learning that directly minimizes the modeling error of the Bayesian classifier without estimating class priors or assuming data separability. By formulating a class-prior-free variational loss based on labeled and unlabeled data distributions, the method enables consistent and efficient optimization, achieving state-of-the-art accuracy on benchmark datasets including CIFAR-10 and Fashion-MNIST.

ABSTRACT

Learning binary classifiers only from positive and unlabeled (PU) data is an important and challenging task in many real-world applications, including web text classification, disease gene identification and fraud detection, where negative samples are difficult to verify experimentally. Most recent PU learning methods are developed based on the conventional misclassification risk of the supervised learning type, and they require to solve the intractable risk estimation problem by approximating the negative data distribution or the class prior. In this paper, we introduce a variational principle for PU learning that allows us to quantitatively evaluate the modeling error of the Bayesian classifier directly from given data. This leads to a loss function which can be efficiently calculated without any intermediate step or model, and a variational learning method can then be employed to optimize the classifier under general conditions. In addition, the discriminative performance and numerical stability of the variational PU learning method can be further improved by incorporating a margin maximizing loss function. We illustrate the effectiveness of the proposed variational method on a number of benchmark examples.

Motivation & Objective

• Address the challenge of training accurate binary classifiers when only positive and unlabeled data are available, especially in scenarios where negative samples are unavailable or hard to verify.
• Overcome the limitations of existing PU learning methods that rely on inaccurate class prior estimation or assume data separability.
• Develop a principled, class-prior-free framework for evaluating and minimizing the modeling error of the Bayesian classifier directly from observed data distributions.
• Enable robust and consistent classifier optimization under general conditions without intermediate estimation steps or strong assumptions.
• Demonstrate superior performance on real-world benchmark datasets, particularly in low-data regimes and imbalanced settings.

Proposed method

• Introduce a variational principle that quantifies the difference between a candidate classifier and the ideal Bayesian classifier using only the empirical distributions of labeled and unlabeled data.
• Define a variational loss function that measures the divergence between the positive data distribution and the classifier's output distribution, avoiding explicit class prior estimation.
• Formulate the optimization problem as minimizing a class-prior-free loss that is consistent and efficiently computable via empirical averaging.
• Utilize adversarial training with a discriminator network to approximate the Jensen-Shannon divergence between the positive and classifier-induced distributions.
• Alternative formulation using a weighted L² distance between the classifier's output and the positive data density, which is minimized via neural network training.
• Apply variational optimization to directly approximate the Bayesian classifier without requiring noisy or heuristic estimates of negative samples or class priors.

Experimental results

Research questions

• RQ1 Can a Bayesian classifier be consistently approximated in PU learning without estimating the class prior or assuming data separability?
• RQ2 How can the modeling error of a classifier be quantitatively evaluated in a class-prior-free manner using only labeled and unlabeled data?
• RQ3 Does a variational loss based on data distributions outperform existing PU learning methods in terms of accuracy and robustness, especially when class priors are unknown?
• RQ4 Can adversarial training with a discriminator effectively minimize the divergence between the positive and classifier distributions in PU learning?
• RQ5 How does the proposed method perform on standard benchmarks like CIFAR-10 and Fashion-MNIST, particularly under data scarcity or class imbalance?

Key findings

• The proposed VPU method achieves state-of-the-art performance on multiple benchmark datasets, including 97.3% accuracy on F-MNIST 1 and 97.6% on STL-10 1, outperforming nnPU, uPU, and GenPU.
• On the Fashion-MNIST dataset, VPU achieves 92.7% accuracy, significantly outperforming nnPU (90.8%) and its MixUp-augmented variant (91.0%), even when the latter uses data augmentation.
• The method maintains high AUC scores across all datasets, with 0.973 AUC on F-MNIST 1 and 0.976 on STL-10 1, indicating strong generalization and robustness.
• VPU is robust to class prior inaccuracies and does not require cross-validation with negative data, as it avoids class prior estimation entirely.
• The variational loss formulation enables consistent optimization without intermediate estimation steps, leading to stable and high-performance results.
• Experiments show that VPU is not sensitive to data augmentation like MixUp, suggesting its inherent robustness compared to methods such as nnPU that benefit less from augmentation.