[Paper Review] Fully Convolutional Multi-Class Multiple Instance Learning

Deepak Pathak, Evan Shelhamer+2 more|arXiv (Cornell University)|Dec 22, 2014

Image Retrieval and Classification TechniquesComputer Science10 references267 citations

TL;DR

This paper proposes a fully convolutional multi-class multiple instance learning (MIL) framework for weakly supervised semantic segmentation using only image-level labels. By jointly optimizing feature representation and pixel-wise classification through a multi-class MIL loss that selects top-scoring pixels per class, the method achieves 25.66% mean intersection-over-union on PASCAL VOC 2012 test set, a 96% relative improvement over baseline models.

ABSTRACT

Multiple instance learning (MIL) can reduce the need for costly annotation in tasks such as semantic segmentation by weakening the required degree of supervision. We propose a novel MIL formulation of multi-class semantic segmentation learning by a fully convolutional network. In this setting, we seek to learn a semantic segmentation model from just weak image-level labels. The model is trained end-to-end to jointly optimize the representation while disambiguating the pixel-image label assignment. Fully convolutional training accepts inputs of any size, does not need object proposal pre-processing, and offers a pixelwise loss map for selecting latent instances. Our multi-class MIL loss exploits the further supervision given by images with multiple labels. We evaluate this approach through preliminary experiments on the PASCAL VOC segmentation challenge.

Motivation & Objective

• To address the high annotation cost of pixel-level semantic segmentation by leveraging only image-level labels.
• To eliminate the need for object proposals or pre-defined instance hypotheses in weakly supervised learning.
• To jointly learn deep feature representations and pixel-wise classifiers via a multi-class MIL loss in a fully convolutional network.
• To improve segmentation accuracy by exploiting inter-class competition in multi-label images.

Proposed method

• The model uses a fully convolutional network (FCN) based on the 16-layer VGG architecture, fine-tuned end-to-end from ImageNet-pretrained weights.
• A multi-class MIL loss is defined by selecting the highest-scoring pixel per class (including background) in the output heat-map for each image.
• The loss is computed only on these selected points, enabling back-propagation while avoiding bias toward background predictions.
• The background class acts as a negative instance, competing with positive classes to refine localization through inter-class confusion.
• Inference involves bilinear interpolation of coarse predictions to full image resolution for pixelwise segmentation.
• The framework avoids instance hypothesis generation and supports variable input sizes without warping or proposal networks.

Experimental results

Research questions

• RQ1 Can end-to-end training of a fully convolutional network jointly optimize representation learning and pixel-level classification under weak supervision?
• RQ2 How does multi-class MIL loss improve localization accuracy compared to single-class or binary MIL in weakly supervised segmentation?
• RQ3 To what extent can inter-class competition in multi-label images enhance the disambiguation of latent object instances?
• RQ4 Can the model avoid degenerate solutions (e.g., predicting all background) without strong supervision or classifier initialization?
• RQ5 How effective is the proposed MIL loss in selecting informative training instances without requiring bounding box annotations?

Key findings

• The MIL-FCN achieved 25.66% mean intersection-over-union (mIoU) on the PASCAL VOC 2012 test set, representing a 96% relative improvement over the baseline with classifier initialization.
• The model converged in under 10,000 iterations with a learning rate of 0.0001, momentum 0.9, and weight decay 0.0005, demonstrating fast and stable training.
• Initialization from ImageNet classifier weights for common classes prevented degenerate solutions and improved performance from 3.52% to 13.11% mIoU on validation.
• The method achieved strong performance without object proposals, data augmentation, or instance-level annotations, relying solely on image-level labels.
• The use of coarse prediction points for loss computation, followed by interpolation, enabled efficient and scalable training and inference.
• Qualitative results in Figure 1 show that the model produces coherent and spatially consistent segmentations, even without ground-truth bounding boxes.