[论文解读] Distribution Aligning Refinery of Pseudo-label for Imbalanced Semi-supervised Learning

Jaehyung Kim, Youngbum Hur+4 more|arXiv (Cornell University)|Jul 17, 2020

Imbalanced Data Classification Techniques参考文献 47被引用 84

一句话总结

本文提出 DARP，一种基于凸优化的方法，在类别不平衡的半监督学习（SSL）中通过使伪标签的分布对齐真实未标记类别分布来 refined，从而提升了现有的 SSL 方法的性能。

ABSTRACT

While semi-supervised learning (SSL) has proven to be a promising way for leveraging unlabeled data when labeled data is scarce, the existing SSL algorithms typically assume that training class distributions are balanced. However, these SSL algorithms trained under imbalanced class distributions can severely suffer when generalizing to a balanced testing criterion, since they utilize biased pseudo-labels of unlabeled data toward majority classes. To alleviate this issue, we formulate a convex optimization problem to softly refine the pseudo-labels generated from the biased model, and develop a simple algorithm, named Distribution Aligning Refinery of Pseudo-label (DARP) that solves it provably and efficiently. Under various class-imbalanced semi-supervised scenarios, we demonstrate the effectiveness of DARP and its compatibility with state-of-the-art SSL schemes.

研究动机与目标

• Motivate and address performance degradation of SSL under imbalanced class distributions where pseudo-labels bias toward majority classes.
• Propose a convex optimization formulation to softly refine pseudo-labels while preserving information from the original predictions.
• Develop an efficient iterative solver (DARP) with convergence guarantees to produce distribution-aligned pseudo-labels.
• Demonstrate DARP's compatibility and improvements across leading SSL schemes on synthetic long-tail datasets and real-world data.

提出的方法

• Formulate a convex optimization problem to minimize weighted KL divergence between refined and original pseudo-labels subject to meeting the true unlabeled class distribution.
• Introduce weights w_m based on entropy to emphasize high-confidence pseudo-labels.
• Provide an efficient dual-coordinate ascent algorithm (Algorithm 1) with provable convergence to the unique solution.
• Enhance refinement with a data-filtering step (removing small entries) to focus on confident signals (Algorithm 2).
• Estimate the true unlabeled class distribution {M_k} via confusion-matrix-based estimation when labeled and unlabeled distributions differ (Section 3.3).

实验结果

研究问题

• RQ1 How does class-imbalance in labeled/unlabeled data bias SSL pseudo-labels and harm minority-class performance?
• RQ2 Can refining pseudo-label distributions to match the true unlabeled distribution improve SSL performance across state-of-the-art methods?
• RQ3 Is it possible to efficiently solve the refinement problem with convergence guarantees while preserving original pseudo-label information?
• RQ4 Does removing small/noisy entries before refinement further enhance the quality of refined pseudo-labels?
• RQ5 How well can the true unlabeled distribution be estimated when it is not directly observable?

主要发现

• DARP consistently improves baseline SSL methods (MixMatch, ReMixMatch, FixMatch) across imbalanced scenarios.
• Refining pseudo-labels to match the true unlabeled distribution yields substantial relative reductions in balanced accuracy and GM under several gamma settings.
• The proposed iterative dual-coordinate ascent solver converges to the unique solution of the refinement problem in practice (T=10 suffices).
• Removing small/noisy pseudo-label entries prior to refinement further enhances performance by biasing refinement toward higher-confidence signals.
• DARP remains effective even when unlabeled and labeled distributions differ (gamma_l ≠ gamma_u) and when unlabeled data are closer to a uniform distribution (e.g., STL-10).
• DARP adds modest computational overhead (at most about 20% of the vanilla SSL run) and is compatible with multiple SSL frameworks.