[Paper Review] Bottom-Up Top-Down Cues for Weakly-Supervised Semantic Segmentation

Qinbin Hou, Puneet K. Dokania+4 more|arXiv (Cornell University)|Dec 7, 2016

Advanced Neural Network Applications4 citations

TL;DR

This paper proposes a weakly-supervised semantic segmentation method using only image-level labels, leveraging bottom-up saliency maps and top-down attention maps to create a highly effective initialization for an EM-based training framework. The approach achieves state-of-the-art performance on PASCAL VOC 2012, outperforming prior methods by a large margin—up to 10% in mIoU—without requiring bounding boxes, scribbles, or CRF post-processing.

ABSTRACT

We consider the task of learning a classifier for semantic segmentation using weak supervision in the form of image labels which specify the object classes present in the image. Our method uses deep convolutional neural networks (CNNs) and adopts an Expectation-Maximization (EM) based approach. We focus on the following three aspects of EM: (i) initialization; (ii) latent posterior estimation (E-step) and (iii) the parameter update (M-step). We show that saliency and attention maps, our bottom-up and top-down cues respectively, of simple images provide very good cues to learn an initialization for the EM-based algorithm. Intuitively, we show that before trying to learn to segment complex images, it is much easier and highly effective to first learn to segment a set of simple images and then move towards the complex ones. Next, in order to update the parameters, we propose minimizing the combination of the standard softmax loss and the KL divergence between the true latent posterior and the likelihood given by the CNN. We argue that this combination is more robust to wrong predictions made by the expectation step of the EM method. We support this argument with empirical and visual results. Extensive experiments and discussions show that: (i) our method is very simple and intuitive; (ii) requires only image-level labels; and (iii) consistently outperforms other weakly-supervised state-of-the-art methods with a very high margin on the PASCAL VOC 2012 dataset.

Motivation & Objective

• Address the challenge of semantic segmentation with minimal supervision, specifically using only image-level labels instead of pixel-level annotations.
• Overcome the limitations of existing weakly-supervised methods that rely on higher supervision such as bounding boxes, points, or superpixels.
• Improve model generalization and convergence by learning from simple images before tackling complex, multi-object scenes.
• Develop a robust parameter update strategy in the M-step that combines softmax loss and KL divergence to mitigate errors from incorrect E-step predictions.

Proposed method

• Use class-agnostic saliency maps (bottom-up) and class-specific attention maps (top-down) from simple ImageNet images to generate a pseudo-ground-truth mask for initial model training.
• Train an initial semantic segmentation model on a filtered subset of ImageNet images with one object per image using the combined saliency and attention map as supervision.
• Apply an Expectation-Maximization (EM) framework: in the E-step, regularize the CNN likelihood using image-level class labels to constrain probability mass to only present classes and background.
• In the M-step, minimize a hybrid loss combining standard softmax cross-entropy and KL divergence between the estimated latent posterior and the CNN’s likelihood to improve robustness.
• Avoid CRF post-processing to maintain full weak supervision; instead, use end-to-end training with only image labels.
• Iteratively refine predictions through multiple EM iterations, starting from the informed initialization, to improve segmentation accuracy on PASCAL VOC 2012.

Experimental results

Research questions

• RQ1 Can a simple, data-efficient initialization using saliency and attention maps significantly improve the performance of weakly-supervised semantic segmentation?
• RQ2 Does combining bottom-up and top-down cues lead to better pseudo-ground-truth supervision than using either cue alone?
• RQ3 Can an EM-based framework with a hybrid M-step loss (softmax + KL divergence) outperform standard softmax-only training in the presence of noisy E-step predictions?
• RQ4 Is it possible to achieve state-of-the-art performance on PASCAL VOC 2012 using only image-level labels without relying on CRF or higher-supervision signals?
• RQ5 How does training on simple images first improve generalization and convergence in weakly-supervised semantic segmentation?

Key findings

• The proposed method achieves a mean Intersection over Union (mIoU) of 72.1% on the PASCAL VOC 2012 test set, outperforming the previous state-of-the-art method (AugFeed) by nearly 10 percentage points.
• Even without CRF post-processing, the method exceeds the best existing method (which uses CRF and higher supervision) by 2.2% on the test set.
• The initial model trained on simple ImageNet images alone achieves better performance than all current state-of-the-art methods, demonstrating the effectiveness of learning from simple images first.
• The combination of saliency and attention maps produces a more accurate pseudo-ground-truth than either cue alone, as shown in visual comparisons and quantitative improvements.
• The hybrid M-step loss (softmax + KL divergence) improves robustness to incorrect E-step predictions, especially in cases of confusing or ambiguous classes.
• The method remains fully weakly-supervised, requiring only image-level labels and no bounding boxes, scribbles, or pixel-level annotations, making it scalable and practical for large-scale data.