[Paper Review] BoxTeacher: Exploring High-Quality Pseudo Labels for Weakly Supervised Instance Segmentation

Tianheng Cheng, Xinggang Wang+3 more|arXiv (Cornell University)|Oct 11, 2022

Advanced Neural Network Applications4 citations

TL;DR

BoxTeacher proposes an end-to-end, self-training framework for weakly supervised instance segmentation that leverages high-quality pseudo masks from box-supervised models. By introducing a mask-aware confidence score and noise-aware loss, it selectively uses reliable pseudo masks to improve student model performance, achieving 35.0 mask AP on COCO with ResNet-50 and 36.5 with ResNet-101—significantly outperforming prior methods and narrowing the gap with fully supervised approaches.

ABSTRACT

Labeling objects with pixel-wise segmentation requires a huge amount of human labor compared to bounding boxes. Most existing methods for weakly supervised instance segmentation focus on designing heuristic losses with priors from bounding boxes. While, we find that box-supervised methods can produce some fine segmentation masks and we wonder whether the detectors could learn from these fine masks while ignoring low-quality masks. To answer this question, we present BoxTeacher, an efficient and end-to-end training framework for high-performance weakly supervised instance segmentation, which leverages a sophisticated teacher to generate high-quality masks as pseudo labels. Considering the massive noisy masks hurt the training, we present a mask-aware confidence score to estimate the quality of pseudo masks and propose the noise-aware pixel loss and noise-reduced affinity loss to adaptively optimize the student with pseudo masks. Extensive experiments can demonstrate the effectiveness of the proposed BoxTeacher. Without bells and whistles, BoxTeacher remarkably achieves 35.0 mask AP and 36.5 mask AP with ResNet-50 and ResNet-101 respectively on the challenging COCO dataset, which outperforms the previous state-of-the-art methods by a significant margin and bridges the gap between box-supervised and mask-supervised methods. The code and models will be available at https://github.com/hustvl/BoxTeacher.

Motivation & Objective

• To investigate whether high-quality pseudo segmentation masks from box-supervised models can be effectively leveraged for weakly supervised instance segmentation.
• To address the limitations of naive self-training, which is hindered by noisy pseudo masks and poor generalization.
• To develop a robust, end-to-end training framework that improves model performance through mutual refinement between a teacher and student.
• To reduce the impact of low-quality pseudo masks using a novel mask-aware confidence score and noise-aware loss functions.

Proposed method

• A teacher network generates high-quality pseudo instance masks and assigns them to ground-truth boxes using a box-based pseudo mask assignment strategy.
• A mask-aware confidence score is introduced to estimate the quality of each pseudo mask, filtering out low-quality predictions.
• The student network is trained with both ground-truth boxes and pseudo masks using a noise-aware pixel loss and a noise-reduced affinity loss.
• Exponential Moving Average (EMA) is applied to progressively update the teacher network based on the student’s evolving weights.
• Strong data augmentation is applied to the student’s input images to improve consistency and robustness.
• The framework is end-to-end and agnostic to the underlying instance segmentation architecture, enabling broad applicability.

Experimental results

Research questions

• RQ1 Can high-quality pseudo segmentation masks generated from box-supervised models be effectively used to improve weakly supervised instance segmentation?
• RQ2 Why does naive self-training with pseudo masks fail to achieve significant gains despite the presence of high-quality masks?
• RQ3 How can the impact of noisy pseudo masks be minimized during self-training in weakly supervised instance segmentation?
• RQ4 Can a collaborative teacher-student framework with confidence scoring and noise-aware losses improve performance beyond standard self-training?
• RQ5 To what extent does consistency regularization via data augmentation enhance learning from noisy pseudo labels?

Key findings

• BoxTeacher achieves 35.0 mask AP on COCO using ResNet-50 and 36.5 with ResNet-101, significantly outperforming previous state-of-the-art methods.
• The proposed mask-aware confidence score leads to better performance than alternative quality estimation methods, including IoU, mean entropy, and IoU-based confidence.
• Naive self-training with pseudo masks performs worse than training with box-only annotations, highlighting the need for noise-aware training strategies.
• BoxTeacher is more robust to strong data augmentation than standard training, especially under longer training schedules.
• The use of strong augmentation improves weakly supervised performance by +0.6 AP but has minimal effect on fully supervised detection, suggesting its role in handling noisy pseudo labels.
• Exponential Moving Average (EMA) has little impact on performance, confirming that the gains in BoxTeacher stem primarily from pseudo labeling and not model averaging.