[Paper Review] SemiReward: A General Reward Model for Semi-supervised Learning

Siyuan Li, Weiyang Jin+5 more|arXiv (Cornell University)|Oct 4, 2023

Advanced Statistical Methods and Models4 citations

TL;DR

SemiReward proposes a general, pluggable reward model that predicts reward scores to evaluate and filter high-quality pseudo-labels in semi-supervised learning, using a two-stage training pipeline with a generator and rewarder network to mitigate confirmation bias. It achieves significant performance gains and faster convergence across 13 diverse SSL benchmarks spanning vision, NLP, and audio tasks, outperforming SOTA methods like Pseudo Label, FlexMatch, and Free/SoftMatch.

ABSTRACT

Semi-supervised learning (SSL) has witnessed great progress with various improvements in the self-training framework with pseudo labeling. The main challenge is how to distinguish high-quality pseudo labels against the confirmation bias. However, existing pseudo-label selection strategies are limited to pre-defined schemes or complex hand-crafted policies specially designed for classification, failing to achieve high-quality labels, fast convergence, and task versatility simultaneously. To these ends, we propose a Semi-supervised Reward framework (SemiReward) that predicts reward scores to evaluate and filter out high-quality pseudo labels, which is pluggable to mainstream SSL methods in wide task types and scenarios. To mitigate confirmation bias, SemiReward is trained online in two stages with a generator model and subsampling strategy. With classification and regression tasks on 13 standard SSL benchmarks across three modalities, extensive experiments verify that SemiReward achieves significant performance gains and faster convergence speeds upon Pseudo Label, FlexMatch, and Free/SoftMatch. Code and models are available at https://github.com/Westlake-AI/SemiReward.

Motivation & Objective

• To address the challenge of confirmation bias in semi-supervised learning caused by low-quality pseudo-labels.
• To develop a general, task-agnostic method for selecting high-quality pseudo-labels without relying on hand-crafted or task-specific policies.
• To enable fast convergence and consistent performance gains across diverse modalities and tasks (classification and regression) in SSL.
• To design a lightweight, pluggable framework that integrates seamlessly with existing SSL methods.
• To train the reward model online with minimal computational overhead using a two-stage pipeline and generator network.

Proposed method

• A rewarder network is trained to predict a smooth, well-calibrated reward score based on the cosine similarity between pseudo-labels and ground-truth labels.
• The rewarder is trained in two stages: first pre-trained on labeled data using a generator to produce 'fake labels', then fine-tuned on a subsampled set of labeled and selected unlabeled data.
• A generator network produces synthetic pseudo-labels during pre-training to decouple rewarder training from the student model and reduce confirmation bias.
• The rewarder is trained to predict ground-truth reward scores using a regression loss, ensuring calibrated and reliable label filtering.
• The framework is plug-and-play, compatible with mainstream SSL methods such as Pseudo Label, FlexMatch, and Free/SoftMatch.
• A subsampling strategy is applied in stage 2 to maintain high sampling rates while filtering out low-quality pseudo-labels based on predicted reward scores.

Experimental results

Research questions

• RQ1 Can a learned reward score based on cosine similarity between pseudo-labels and ground-truth labels serve as a reliable, general-purpose metric for pseudo-label quality in both classification and regression tasks?
• RQ2 How can a reward model be trained online with minimal computational cost while avoiding confirmation bias in self-training SSL?
• RQ3 To what extent can a pluggable reward module improve performance and convergence speed across diverse SSL benchmarks and modalities?
• RQ4 Does the two-stage training pipeline with a generator and rewarder effectively decouple rewarder training from the student model and improve generalization?
• RQ5 Can the rewarder generalize across different SSL algorithms and datasets without requiring task-specific adaptation?

Key findings

• SemiReward achieves performance gains of +1.9% to +3.7% accuracy over baseline SSL methods on 13 standard SSL benchmarks across vision, NLP, and audio modalities.
• The method reduces the number of training iterations required to reach peak performance by at least 1.7 times compared to baselines, indicating faster convergence.
• On CV, NLP, and audio datasets, SemiReward consistently improves SOTA methods including Pseudo Label, FlexMatch, and Free/SoftMatch across both classification and regression tasks.
• The reward scores predicted by SemiReward are well-calibrated and reliably identify high-quality pseudo-labels, as validated through empirical analysis.
• The two-stage training pipeline with a generator effectively mitigates confirmation bias, enabling stable and accurate reward model training.
• The framework is highly generalizable and achieves consistent improvements across diverse datasets and task types without requiring architectural or hyperparameter changes.