[Paper Review] Revisiting Sample Selection Approach to Positive-Unlabeled Learning: Turning Unlabeled Data into Positive rather than Negative

Miao Xu, Bingcong Li+3 more|arXiv (Cornell University)|Jan 29, 2019

Machine Learning and Data Classification38 references4 citations

TL;DR

This paper proposes aaPU, a novel positive-unlabeled (PU) learning method that selects high-loss unlabeled samples as pseudo-positive data during training, leveraging deep network memorization properties. By reweighting these biased pseudo-positive samples through a new learning objective, aaPU outperforms state-of-the-art importance reweighting methods like nnPU on CIFAR-10, CIFAR-100, and 20Newsgroups, demonstrating that sample selection can surpass importance reweighting in deep learning settings.

ABSTRACT

In the early history of positive-unlabeled (PU) learning, the sample selection approach, which heuristically selects negative (N) data from U data, was explored extensively. However, this approach was later dominated by the importance reweighting approach, which carefully treats all U data as N data. May there be a new sample selection method that can outperform the latest importance reweighting method in the deep learning age? This paper is devoted to answering this question affirmatively---we propose to label large-loss U data as P, based on the memorization properties of deep networks. Since P data selected in such a way are biased, we develop a novel learning objective that can handle such biased P data properly. Experiments confirm the superiority of the proposed method.

Motivation & Objective

• To investigate whether a modern sample selection approach can outperform importance reweighting in deep PU learning.
• To address the challenge of biased pseudo-positive data selected from unlabeled sets using deep networks.
• To develop a new learning objective that properly handles the bias introduced by selecting high-loss samples as positive.
• To demonstrate the superiority of this approach over existing importance reweighting methods like nnPU in real-world and synthetic datasets.

Proposed method

• The method selects unlabeled samples with the highest cross-entropy loss (using -1 as ground truth) as pseudo-positive data, based on deep network memorization properties.
• It introduces a novel learning objective that accounts for the bias in the selected pseudo-positive samples, avoiding underestimation of empirical risk.
• The selected pseudo-positive data are permanently added to the training set in subsequent epochs, starting after validation loss stabilizes.
• The approach uses a dynamic data addition strategy, adding 150, 160, or 320 instances per epoch depending on the dataset and model scale.
• It employs standard deep learning architectures (e.g., ResNet-like for CIFAR, MLP for 20Newsgroups) with adaptive learning rate and weight decay schedules.
• The method is end-to-end and operates in an online fashion, updating the training set with new pseudo-positives after each epoch.

Experimental results

Research questions

• RQ1 Can a sample selection approach that selects positive data from unlabeled sets outperform modern importance reweighting methods in deep PU learning?
• RQ2 How can high-loss unlabeled samples be reliably identified as pseudo-positive data using deep network behavior?
• RQ3 What learning objective is needed to effectively handle the inherent bias in pseudo-positive samples selected from unlabeled data?
• RQ4 At what training stage and with what frequency should pseudo-positive samples be added to maximize performance?

Key findings

• aaPU achieves superior performance compared to nnPU on all three benchmark datasets: CIFAR-10, CIFAR-100, and 20Newsgroups.
• On CIFAR-100 and 20Newsgroups, the performance gain of aaPU over nnPU is particularly significant, with improvements observed after 60–100 epochs.
• The method shows consistent improvement across 10 repeated experiments, with reported average accuracy and variance indicating robustness.
• The performance gain emerges only after a sufficient number of epochs, suggesting that the method benefits from stable model representations before pseudo-positive data are added.
• The optimal number of pseudo-positives to add per epoch lies in the range [40, 80, 160, 320], with 160–320 yielding the best results on larger datasets.
• The method is effective even when using complex deep networks, overcoming the overfitting issues that plague traditional sample selection in deep learning.