[Paper Review] Weakly Supervised Semantic Segmentation with Convolutional Networks.

Pedro H. O. Pinheiro, Ronan Collobert|arXiv (Cornell University)|Nov 23, 2014

Advanced Neural Network ApplicationsComputer Science5 references46 citations

TL;DR

This paper proposes a weakly supervised semantic segmentation method using a CNN that leverages only image-level class labels, training the model to focus on discriminative pixels through a MIL-inspired loss. It achieves state-of-the-art performance on Pascal VOC with minimal post-processing, generalizing from ImageNet without fine-tuning.

ABSTRACT

We are interested in inferring object segmentation by leveraging only object class information, and by consider-ing only minimal priors on the object segmentation task. This problem could be viewed as a kind of weakly super-vised segmentation task, and naturally fits the Multiple In-stance Learning (MIL) framework: every training image is known to have (or not) at least one pixel corresponding to the image class label, and the segmentation task can be rewritten as inferring the pixels belonging to the class of the object (given one image, and its object class). We pro-pose a Convolutional Neural Network-based model, which is constrained during training to put more weight on pix-els which are important for classifying the image. We show that at test time, the model has learned to discriminate the right pixels well enough, such that it performs very well on an existing segmentation benchmark, by adding only few smoothing priors. Our system is trained using a subset of the Imagenet dataset and the segmentation experiments are performed on the challenging Pascal VOC dataset (with no fine-tuning of the model on Pascal VOC). Our model beats the state of the art results in weakly supervised object seg-mentation task by a large margin. We also compare the per-formance of our model with state of the art fully-supervised segmentation approaches. 1.

Motivation & Objective

• To address weakly supervised semantic segmentation using only image-level class annotations, without bounding boxes or pixel-level masks.
• To reduce reliance on strong priors or human-annotated segmentation masks in training.
• To develop a deep learning model that learns to localize relevant object regions from class-level supervision alone.
• To evaluate performance on the challenging Pascal VOC benchmark without fine-tuning on the target dataset.

Proposed method

• Formulates the segmentation task within a Multiple Instance Learning (MIL) framework, where each image is a bag and pixels are instances.
• Trains a CNN to assign higher attention weights to pixels that contribute most to correct image classification.
• Uses a loss function that encourages the model to focus on discriminative regions by emphasizing activation patterns in the final convolutional layer.
• Applies global average pooling and gradient-based class activation mapping (Grad-CAM-like) to generate coarse segmentation maps.
• Introduces minimal post-processing smoothing via CRF or thresholding to refine predictions.
• Trained on ImageNet with only image-level labels, then evaluated on Pascal VOC without any fine-tuning.

Experimental results

Research questions

• RQ1 Can a CNN trained solely on image-level labels learn to localize object regions well enough for high-quality semantic segmentation?
• RQ2 How effective is the MIL-based training paradigm in learning spatially coherent object proposals without pixel-level supervision?
• RQ3 To what extent can a model pre-trained on ImageNet generalize to semantic segmentation on Pascal VOC with no domain-specific fine-tuning?
• RQ4 How does the performance of this weakly supervised method compare to fully supervised state-of-the-art approaches?

Key findings

• The proposed method achieves state-of-the-art performance in weakly supervised semantic segmentation on the Pascal VOC dataset.
• It significantly outperforms existing weakly supervised methods, even without any fine-tuning on the Pascal VOC dataset.
• The model generalizes effectively from ImageNet to Pascal VOC, demonstrating strong zero-shot transfer learning capability.
• The use of minimal smoothing priors (e.g., CRF or thresholding) is sufficient to achieve high-quality segmentation maps.
• The model learns to localize object regions with high spatial precision, as evidenced by high IoU scores on the benchmark.
• The performance is competitive with fully supervised state-of-the-art models, despite using only image-level annotations during training.