[Paper Review] Dist-PU: Positive-Unlabeled Learning from a Label Distribution Perspective

Yunrui Zhao, Qianqian Xu+3 more|arXiv (Cornell University)|Dec 6, 2022

Machine Learning and Data Classification5 citations

TL;DR

This paper proposes Dist-PU, a novel positive-unlabeled (PU) learning method that aligns the expected label distribution of model predictions with the ground-truth label distribution derived from the known class prior. By leveraging label distribution consistency, entropy minimization, and Mixup regularization, Dist-PU mitigates the negative-prediction bias common in cost-sensitive PU methods and achieves state-of-the-art performance on CIFAR-10, Fashion-MNIST, and Alzheimer's disease datasets.

ABSTRACT

Positive-Unlabeled (PU) learning tries to learn binary classifiers from a few labeled positive examples with many unlabeled ones. Compared with ordinary semi-supervised learning, this task is much more challenging due to the absence of any known negative labels. While existing cost-sensitive-based methods have achieved state-of-the-art performances, they explicitly minimize the risk of classifying unlabeled data as negative samples, which might result in a negative-prediction preference of the classifier. To alleviate this issue, we resort to a label distribution perspective for PU learning in this paper. Noticing that the label distribution of unlabeled data is fixed when the class prior is known, it can be naturally used as learning supervision for the model. Motivated by this, we propose to pursue the label distribution consistency between predicted and ground-truth label distributions, which is formulated by aligning their expectations. Moreover, we further adopt the entropy minimization and Mixup regularization to avoid the trivial solution of the label distribution consistency on unlabeled data and mitigate the consequent confirmation bias. Experiments on three benchmark datasets validate the effectiveness of the proposed method.Code available at: https://github.com/Ray-rui/Dist-PU-Positive-Unlabeled-Learning-from-a-Label-Distribution-Perspective.

Motivation & Objective

• To address the negative-prediction preference in existing cost-sensitive PU learning methods, which arises from explicitly minimizing the risk of classifying unlabeled data as negative.
• To propose a new perspective for PU learning by modeling the label distribution of unlabeled data as a fixed, learnable supervision signal.
• To improve generalization and reduce confirmation bias by integrating entropy minimization and Mixup regularization into the label distribution alignment framework.
• To provide theoretical justification through a generalization bound for label distribution alignment in PU learning.

Proposed method

• Formulate PU learning as a label distribution alignment task by aligning the expectation of predicted label distributions with the ground-truth class prior.
• Estimate the expected predicted label via sigmoid outputs from a deep neural network, enabling end-to-end training.
• Introduce entropy minimization to encourage confident predictions (scores near 0 or 1), preventing trivial solutions where all unlabeled predictions equal the class prior.
• Apply Mixup regularization to data augmentation, reducing confirmation bias by encouraging robustness to interpolated samples.
• Combine label distribution alignment, entropy minimization, and Mixup into a unified training objective for improved generalization.
• Derive a generalization bound for label distribution alignment to provide theoretical support for the method's effectiveness.

Experimental results

Research questions

• RQ1 Can label distribution alignment between predicted and ground-truth distributions effectively mitigate the negative-prediction bias in PU learning?
• RQ2 How does entropy minimization improve the robustness of label distribution alignment and prevent trivial solutions?
• RQ3 To what extent does Mixup regularization reduce confirmation bias in PU learning when combined with label distribution alignment?
• RQ4 How do hyperparameters such as μ, α, ν, and γ affect the performance and stability of the Dist-PU framework?
• RQ5 Does the proposed label distribution alignment framework generalize to real-world datasets with known class priors?

Key findings

• Dist-PU outperforms state-of-the-art PU learning methods on all three benchmark datasets—CIFAR-10, Fashion-MNIST, and Alzheimer’s disease—across most evaluation metrics.
• Abllation studies confirm that label distribution alignment is the most critical component, significantly improving performance over baselines without it.
• Entropy minimization provides a minor but consistent improvement, especially when combined with label distribution alignment.
• Mixup regularization shows the most substantial gain when used in conjunction with label distribution alignment, highlighting its synergy in reducing confirmation bias.
• Sensitivity analysis shows that μ > 0.5 degrades performance, while α > 1 and ν > 1 improve robustness and reduce confirmation bias.
• Visualization of wrong predictions shows that Mixup leads to smoother score distributions and lower confidence in incorrect predictions, confirming its role in mitigating confirmation bias.