[Paper Review] One-bit Supervision for Image Classification

Hengtong Hu, Lingxi Xie+3 more|arXiv (Cornell University)|Sep 14, 2020

Domain Adaptation and Few-Shot Learning39 references4 citations

TL;DR

This paper introduces one-bit supervision, a novel semi-supervised learning setting where models query predicted labels and receive only a yes/no response per sample, drastically reducing annotation cost. By using a multi-stage training framework with negative label suppression, the method improves guess accuracy over time and effectively utilizes incorrect guesses, achieving higher accuracy than standard semi-supervised learning under the same supervision budget on CIFAR100, Mini-ImageNet, and ImageNet.

ABSTRACT

This paper presents one-bit supervision, a novel setting of learning from incomplete annotations, in the scenario of image classification. Instead of training a model upon the accurate label of each sample, our setting requires the model to query with a predicted label of each sample and learn from the answer whether the guess is correct. This provides one bit (yes or no) of information, and more importantly, annotating each sample becomes much easier than finding the accurate label from many candidate classes. There are two keys to training a model upon one-bit supervision: improving the guess accuracy and making use of incorrect guesses. For these purposes, we propose a multi-stage training paradigm which incorporates negative label suppression into an off-the-shelf semi-supervised learning algorithm. In three popular image classification benchmarks, our approach claims higher efficiency in utilizing the limited amount of annotations.

Motivation & Objective

• To reduce the high annotation cost in large-class image classification by replacing full-label annotation with simple yes/no feedback.
• To address the challenge of learning efficiently from sparse, weak supervision where only one bit of information (correct/incorrect) is provided per sample.
• To improve model performance by increasing the accuracy of model-generated guesses over multiple training stages.
• To effectively utilize incorrect guesses by suppressing predicted classes in the loss function, turning negative feedback into useful training signals.
• To demonstrate that one-bit supervision can achieve higher learning efficiency than traditional semi-supervised learning under equivalent supervision bits.

Proposed method

• The method employs a multi-stage training paradigm where the model iteratively queries its predictions on unlabeled data.
• In each stage, the model makes predictions on a random subset of unlabeled images and receives one-bit feedback (correct or incorrect).
• Correct guesses are added to the supervised set, while incorrect guesses are used to suppress the predicted class in the loss function.
• Negative label suppression is applied by minimizing the model's confidence on the incorrectly predicted class, effectively turning a wrong prediction into a weak negative signal.
• The process starts with a small set of accurate labels and uses a pre-trained model (e.g., Mean-Teacher) as the baseline for semi-supervised learning.
• The model is retrained at each stage with the expanded labeled set and the suppressed negative signals, gradually improving prediction accuracy.

Experimental results

Research questions

• RQ1 Can one-bit supervision, where only a yes/no response is given per prediction, achieve higher learning efficiency than standard semi-supervised learning?
• RQ2 How can incorrect guesses in one-bit supervision be effectively leveraged as training signals rather than discarded?
• RQ3 Does a multi-stage training framework that progressively improves guess accuracy lead to better model performance?
• RQ4 Can negative label suppression significantly enhance model learning from weak supervision in high-class image classification tasks?
• RQ5 How does one-bit supervision compare to full-label semi-supervised learning when both are constrained by the same number of supervision bits?

Key findings

• One-bit supervision reduced average annotation time to 2.72 seconds per image (92.3% precision), compared to ~1 minute for full-label annotation, validating its efficiency in large-class settings.
• The proposed multi-stage framework with negative label suppression achieved higher accuracy than standard semi-supervised learning under the same number of supervision bits on CIFAR100, Mini-ImageNet, and ImageNet.
• On CIFAR100 and Mini-ImageNet, using 20% accurate labels plus one-bit feedback outperformed semi-supervised learning with 100% accurate labels in terms of learning efficiency.
• Diagnostic experiments confirmed that the performance gain stems from more effective use of incomplete supervision, particularly through improved guess accuracy and negative label utilization.
• The method demonstrated robustness across datasets with varying numbers of classes, showing strong potential for real-world deployment in data-scarce scenarios.
• The framework generalizes to other vision tasks, suggesting applicability to object detection and semantic segmentation with similar weak supervision paradigms.