[Paper Review] Salvage of Supervision in Weakly Supervised Object Detection

Sui Lin, Chen-Lin Zhang+1 more|arXiv (Cornell University)|Jun 8, 2021

Advanced Neural Network Applications4 citations

TL;DR

This paper proposes Salvage of Supervision (SoS-WSOD), a three-stage framework that enhances weakly supervised object detection by leveraging image-level labels, generating high-quality pseudo-box annotations, and applying semi-supervised learning to noisy pseudo-labeled data. By integrating modern fully supervised detection techniques like ResNet, RoIAlign, and FPN, SoS-WSOD achieves state-of-the-art performance with 64.4 mAP50 on VOC2007, 61.9 mAP50 on VOC2012, and 16.6 mAP50:95 on MS-COCO, while also enabling faster inference by eliminating external proposal generation.

ABSTRACT

Weakly supervised object detection~(WSOD) has recently attracted much attention. However, the lack of bounding-box supervision makes its accuracy much lower than fully supervised object detection (FSOD), and currently modern FSOD techniques cannot be applied to WSOD. To bridge the performance and technical gaps between WSOD and FSOD, this paper proposes a new framework, Salvage of Supervision (SoS), with the key idea being to harness every potentially useful supervisory signal in WSOD: the weak image-level labels, the pseudo-labels, and the power of semi-supervised object detection. This paper proposes new approaches to utilize these weak and noisy signals effectively, and shows that each type of supervisory signal brings in notable improvements, outperforms existing WSOD methods (which mainly use only the weak labels) by large margins. The proposed SoS-WSOD method also has the ability to freely use modern FSOD techniques. SoS-WSOD achieves 64.4 $m ext{AP}_{50}$ on VOC2007, 61.9 $m ext{AP}_{50}$ on VOC2012 and 16.6 $m ext{AP}_{50:95}$ on MS-COCO, and also has fast inference speed. Ablations and visualization further verify the effectiveness of SoS.

Motivation & Objective

• To close the performance and technical gap between weakly supervised object detection (WSOD) and fully supervised object detection (FSOD), which currently limits WSOD to outdated architectures and slower inference.
• To address the underutilization of supervisory signals in WSOD, particularly the lack of effective use of pseudo-box annotations and unlabeled data.
• To enable WSOD to fully benefit from modern FSOD techniques—such as ResNet backbones, RoIAlign, and FPN—by developing a framework that can effectively harness pseudo-supervision and noisy-label learning.
• To improve WSOD accuracy and speed by introducing a novel three-stage pipeline that systematically recovers and reuses supervisory signals.

Proposed method

• Stage 1 trains a weakly supervised detector using image-level labels, employing an improved OICR-based method with a new post-processing module (PGF) to filter and refine detection results.
• Stage 2 generates high-quality pseudo-box annotations via the PGF module, enabling a pseudo-FSOD training phase that leverages modern FSOD techniques like ResNet, RoIAlign, and FPN.
• Stage 3 introduces a novel semi-supervised object detection (SSOD) setup by splitting the training set into 'clean' (labeled) and 'noisy' (unlabeled) subsets, using a rule-of-thumb to balance class distribution and reduce noise impact.
• The PGF module uses two hyperparameters, $t_{keep}$ and $t_{con}$, to filter low-confidence detections and refine pseudo-annotations based on confidence and consistency across proposals.
• The framework treats the entire pseudo-labeled dataset as noisy, applying semi-supervised learning to improve generalization and reduce error propagation from imperfect pseudo-labels.
• The method avoids external object proposal generation, enabling faster inference by relying on end-to-end detection heads.

Experimental results

Research questions

• RQ1 Can modern fully supervised object detection techniques—such as ResNet, RoIAlign, and FPN—be effectively applied in weakly supervised object detection when only image-level labels are available?
• RQ2 How can high-quality pseudo-box annotations be generated from weakly supervised detectors to enable effective fine-tuning with fully supervised methods?
• RQ3 Can semi-supervised object detection be effectively leveraged in WSOD by treating pseudo-labeled data as noisy, and what splitting strategy maximizes performance?
• RQ4 What is the optimal balance between labeled and unlabeled data in the semi-supervised stage to avoid class imbalance and performance collapse?
• RQ5 To what extent can the performance of WSOD be improved by systematically salvaging all available supervisory signals, including weak labels, pseudo-labels, and unlabeled data?

Key findings

• SoS-WSOD achieves 64.4 mAP50 on VOC2007, significantly outperforming prior WSOD methods that rely only on image-level labels.
• On VOC2012, SoS-WSOD reaches 61.9 mAP50, demonstrating strong generalization across datasets despite the absence of box-level annotations.
• On MS-COCO, SoS-WSOD attains 16.6 mAP50:95, marking a substantial improvement over previous WSOD approaches and closing the gap with fully supervised detectors.
• The framework achieves faster inference than baseline WSOD methods, with an inference time of 0.031 seconds per image on a single RTX3090 GPU, eliminating the need for time-consuming external proposal generation.
• Ablation studies confirm that each stage—pseudo-FSOD and semi-supervised learning—contributes significantly to performance gains, with the semi-supervised stage being particularly effective when using an optimal split size (K=2000 on VOC2007).
• Visualization results show that SoS-WSOD detects more accurate and complete objects, even in complex scenes, and effectively reduces issues like part domination and clustered predictions.