[Paper Review] Weakly- and Semi-Supervised Learning of a DCNN for Semantic Image Segmentation

George Papandreou, Liang-Chieh Chen+2 more|arXiv (Cornell University)|Feb 9, 2015

Advanced Neural Network ApplicationsComputer Science31 references361 citations

TL;DR

This paper proposes an Expectation-Maximization (EM) framework for training DeepLab-CRF models using weakly supervised (image-level labels or bounding boxes) and semi-supervised (few pixel-level + many weakly labeled images) data. It achieves 69.0% mean IOU on PASCAL VOC 2012 using only weak annotations, and 73.9% by combining PASCAL and MS-COCO datasets, significantly reducing annotation cost while matching fully supervised performance.

ABSTRACT

Deep convolutional neural networks (DCNNs) trained on a large number of images with strong pixel-level annotations have recently significantly pushed the state-of-art in semantic image segmentation. We study the more challenging problem of learning DCNNs for semantic image segmentation from either (1) weakly annotated training data such as bounding boxes or image-level labels or (2) a combination of few strongly labeled and many weakly labeled images, sourced from one or multiple datasets. We develop Expectation-Maximization (EM) methods for semantic image segmentation model training under these weakly supervised and semi-supervised settings. Extensive experimental evaluation shows that the proposed techniques can learn models delivering competitive results on the challenging PASCAL VOC 2012 image segmentation benchmark, while requiring significantly less annotation effort. We share source code implementing the proposed system at https://bitbucket.org/deeplab/deeplab-public.

Motivation & Objective

• To reduce the high annotation cost of pixel-level segmentation by leveraging weakly supervised and semi-supervised learning.
• To develop an EM-based method that trains DeepLab-CRF models using image-level labels or bounding boxes as weak supervision.
• To evaluate performance in semi-supervised settings where only a small number of pixel-level annotated images are available alongside large amounts of weakly labeled data.
• To improve performance by combining annotations from multiple datasets, such as PASCAL and MS-COCO.
• To demonstrate that weak supervision can yield competitive results with significantly less annotation effort.

Proposed method

• The method uses an Expectation-Maximization (EM) algorithm that alternates between estimating latent pixel-wise labels under weak supervision constraints and optimizing DeepLab-CRF model parameters via stochastic gradient descent (SGD).
• In the E-step, the model infers per-pixel predictions conditioned on weak labels (e.g., image-level class presence or bounding box regions), enforcing consistency with the weak supervision.
• In the M-step, the DCNN and CRF parameters are updated using standard backpropagation on the estimated pixel-level labels.
• For bounding box supervision, the method uses foreground/background segmentation from bounding boxes as weak supervision, avoiding reliance on external objectness or segmentation modules.
• The approach is extended to semi-supervised learning by combining a small set of pixel-level annotated images with a large pool of weakly labeled images (bounding boxes or image-level labels).
• The method is further enhanced by multi-dataset pretraining and joint training using strong annotations from both PASCAL and MS-COCO.

Experimental results

Research questions

• RQ1 Can a DCNN-based semantic segmentation model achieve competitive performance when trained solely on image-level labels without any pixel-level annotations?
• RQ2 How effective is the proposed EM-based method for training semantic segmentation models using only bounding box annotations?
• RQ3 To what extent can a small number of pixel-level annotated images combined with a large number of weakly labeled images match the performance of a fully supervised model?
• RQ4 Can combining weak or strong annotations from multiple datasets (e.g., PASCAL and MS-COCO) further improve segmentation accuracy?
• RQ5 Does the proposed EM framework outperform existing MIL-based methods in weakly supervised semantic segmentation?

Key findings

• The model achieves 62.2% mean IOU on PASCAL VOC 2012 using only bounding box annotations, demonstrating strong performance with minimal supervision.
• With image-level labels alone, the method achieves 39.6% IOU, approaching state-of-the-art results without external objectness or segmentation modules.
• In the semi-supervised setting, using 2.9k pixel-level and 9k image-level annotated images yields 68.5% IOU, only 2% below the fully supervised baseline.
• By combining strong annotations from PASCAL and MS-COCO datasets, the model reaches 73.9% IOU on PASCAL VOC 2012, surpassing previous methods.
• The EM-based approach outperforms existing MIL-based methods and achieves near-state-of-the-art performance with significantly reduced annotation cost.
• The method generalizes well across datasets and settings, showing robustness and scalability in weakly and semi-supervised learning scenarios.