[Paper Review] End-to-End Semi-Supervised Object Detection with Soft Teacher

Mengde Xu, Zheng Zhang+6 more|arXiv (Cornell University)|Jun 16, 2021

Advanced Neural Network Applications4 citations

TL;DR

This paper proposes an end-to-end semi-supervised object detection framework using a soft teacher and box jittering to improve pseudo-label quality and detection performance. By jointly training a student detector and a teacher model via exponential moving average, and weighting classification loss by teacher confidence scores, the method achieves state-of-the-art results on COCO, improving a 40.9 mAP detector to 44.5 mAP with 123K unlabeled images and boosting a Swin Transformer-based detector to 61.3 mAP on test-dev2017.

ABSTRACT

This paper presents an end-to-end semi-supervised object detection approach, in contrast to previous more complex multi-stage methods. The end-to-end training gradually improves pseudo label qualities during the curriculum, and the more and more accurate pseudo labels in turn benefit object detection training. We also propose two simple yet effective techniques within this framework: a soft teacher mechanism where the classification loss of each unlabeled bounding box is weighed by the classification score produced by the teacher network; a box jittering approach to select reliable pseudo boxes for the learning of box regression. On the COCO benchmark, the proposed approach outperforms previous methods by a large margin under various labeling ratios, i.e. 1\%, 5\% and 10\%. Moreover, our approach proves to perform also well when the amount of labeled data is relatively large. For example, it can improve a 40.9 mAP baseline detector trained using the full COCO training set by +3.6 mAP, reaching 44.5 mAP, by leveraging the 123K unlabeled images of COCO. On the state-of-the-art Swin Transformer based object detector (58.9 mAP on test-dev), it can still significantly improve the detection accuracy by +1.5 mAP, reaching 60.4 mAP, and improve the instance segmentation accuracy by +1.2 mAP, reaching 52.4 mAP. Further incorporating with the Object365 pre-trained model, the detection accuracy reaches 61.3 mAP and the instance segmentation accuracy reaches 53.0 mAP, pushing the new state-of-the-art.

Motivation & Objective

• To address the limitations of multi-stage semi-supervised object detection methods that rely on inaccurate initial pseudo-labels.
• To improve pseudo-label quality during training by enabling mutual refinement between detection and pseudo-labeling.
• To develop a simple yet effective end-to-end framework that avoids complex training pipelines.
• To leverage unlabeled data more effectively through confidence-weighted supervision and robust box selection.
• To achieve state-of-the-art performance under low and high labeling ratios, including full COCO training set.

Proposed method

• The framework uses a student model for detection training and a teacher model, updated via exponential moving average (EMA), for online pseudo-labeling of unlabeled images.
• A soft teacher mechanism weights the classification loss of each unlabeled box by the teacher's confidence score, enabling more informative supervision.
• Box jittering is applied to estimate regression variance across multiple perturbed versions of predicted boxes, improving selection of reliable regression targets.
• Pseudo-labels are filtered using a high foreground threshold (0.9) to ensure precision, while background boxes are weighted by teacher confidence to reduce false negatives.
• The final loss combines supervised detection loss on labeled data and unsupervised detection loss on pseudo-labeled unlabeled data.
• The end-to-end training enables a flywheel effect where improved detection leads to better pseudo-labels, which in turn improve detection.

Figure 1: The proposed end-to-end pseudo-label based semi-supervised object detection method outperforms the STAC [ 27 ] by a large margin on MS-COCO benchmark.

Experimental results

Research questions

• RQ1 Can an end-to-end training framework outperform multi-stage approaches in semi-supervised object detection?
• RQ2 How does soft teacher supervision, based on confidence scores, improve pseudo-label quality and detection accuracy?
• RQ3 What is the impact of box jittering on selecting reliable regression targets in semi-supervised object detection?
• RQ4 Can the proposed method significantly improve performance even when full labeled data is used, by leveraging large-scale unlabeled data?
• RQ5 How does the framework perform across different detector architectures, including state-of-the-art Swin Transformer models?

Key findings

• The proposed method improves a 40.9 mAP detector trained on the full COCO training set to 44.5 mAP by leveraging 123K unlabeled images, achieving a +3.6 mAP gain.
• On the state-of-the-art HTC++(Swin-L) detector, the method boosts mAP from 58.9 to 60.4 on test-dev2017, surpassing 60 mAP for the first time.
• With Object365 pre-training, the method achieves 61.3 mAP on detection and 53.0 mAP on instance segmentation, setting a new SOTA on COCO.
• The ablation study shows that soft teacher and box jittering together improve mAP by 3.0 points over EMA-only baseline, with soft teacher alone contributing 2.4 points.
• The optimal foreground threshold for pseudo-label filtering is 0.9, and the best box jittering count is 10, with performance saturated at N=10.
• The method outperforms previous SOTA methods like STAC and self-training, even with significantly less unlabeled data (123K vs. 2.9M images).

Figure 2: The overview of the end-to-end pseudo-labeling framework for semi-supervised object detection. Unlabeled images and labeled images form the training data batch. In each training iteration, a soft teacher is applied to perform pseudo-labeling on weak augmented unlabeled images on the fly. T