[Paper Review] BoxSup: Exploiting Bounding Boxes to Supervise Convolutional Networks for Semantic Segmentation

Jifeng Dai, Kaiming He+1 more|arXiv (Cornell University)|Mar 5, 2015

Advanced Neural Network ApplicationsComputer Science33 references146 citations

TL;DR

BoxSup proposes a weakly supervised semantic segmentation method that trains deep convolutional networks using only bounding box annotations, leveraging iterative refinement between region proposals and model updates. It achieves state-of-the-art performance on PASCAL VOC 2012 (40.5 mean IoU) and PASCAL-CONTEXT (40.5 mean IoU), outperforming fully mask-supervised baselines by exploiting large-scale COCO bounding boxes, demonstrating that coarse annotations can effectively boost segmentation accuracy when combined with iterative optimization.

ABSTRACT

Recent leading approaches to semantic segmentation rely on deep convolutional networks trained with human-annotated, pixel-level segmentation masks. Such pixel-accurate supervision demands expensive labeling effort and limits the performance of deep networks that usually benefit from more training data. In this paper, we propose a method that achieves competitive accuracy but only requires easily obtained bounding box annotations. The basic idea is to iterate between automatically generating region proposals and training convolutional networks. These two steps gradually recover segmentation masks for improving the networks, and vise versa. Our method, called BoxSup, produces competitive results supervised by boxes only, on par with strong baselines fully supervised by masks under the same setting. By leveraging a large amount of bounding boxes, BoxSup further unleashes the power of deep convolutional networks and yields state-of-the-art results on PASCAL VOC 2012 and PASCAL-CONTEXT.

Motivation & Objective

• To reduce reliance on expensive, pixel-level segmentation masks in training deep CNNs for semantic segmentation.
• To explore whether bounding box annotations—readily available in large quantities—can serve as a viable alternative or complement to mask supervision.
• To develop an iterative training framework that progressively improves segmentation masks from coarse box-level supervision.
• To demonstrate that large-scale bounding box data can significantly enhance model performance, even surpassing fully mask-supervised baselines when combined with weak supervision.

Proposed method

• Iteratively alternates between generating region proposals from a trained network and retraining the network using these proposals as pseudo-masks.
• Uses unsupervised region proposal methods (e.g., Selective Search) to generate candidate segmentation masks from bounding box constraints.
• Trains a fully convolutional network (FCN) using the generated pseudo-masks, with loss computed via cross-entropy or IoU-based optimization.
• Fine-tunes the network on a combination of COCO bounding boxes and limited PASCAL VOC mask annotations to improve generalization.
• Applies CRF post-processing and test-time data augmentation (multi-scale inference) to further boost performance.
• Leverages pre-training on ImageNet and transfer learning to improve feature quality even with weak supervision.

Experimental results

Research questions

• RQ1 Can semantic segmentation models achieve competitive performance when trained solely on bounding box annotations instead of pixel-level masks?
• RQ2 To what extent can large-scale bounding box annotations improve the performance of weakly supervised semantic segmentation models?
• RQ3 Does the iterative refinement between region proposals and network updates lead to progressively better segmentation masks and model accuracy?
• RQ4 Can bounding box supervision outperform or match the performance of fully mask-supervised models when combined with large-scale data?

Key findings

• BoxSup achieves 40.5 mean IoU on PASCAL-CONTEXT using only 10k mask annotations and 133k COCO bounding boxes, outperforming a fully mask-supervised baseline (35.7 mean IoU).
• On PASCAL VOC 2012, BoxSup reaches 73.1 mAP using VOC masks and COCO bounding boxes, surpassing the 71.0 mAP of a strong baseline using only COCO masks.
• The method achieves state-of-the-art performance on both PASCAL VOC 2012 and PASCAL-CONTEXT benchmarks, even with significantly weaker supervision than standard mask-based methods.
• A semi-supervised variant replacing 9/10 mask annotations with bounding boxes yields comparable accuracy to fully mask-supervised models, indicating substantial labeling cost reduction.
• Error analysis shows that the primary benefit of bounding box data is improved object recognition accuracy, with secondary gains in boundary localization.
• Test-time scale augmentation boosts BoxSup performance from 73.1 to 75.2 mAP on VOC 2012, approaching the performance of strong mask-supervised models.