[Paper Review] Learning Pixel-level Semantic Affinity with Image-level Supervision for Weakly Supervised Semantic Segmentation

Jiwoon Ahn, Suha Kwak|arXiv (Cornell University)|Mar 28, 2018

Advanced Neural Network Applications40 references74 citations

TL;DR

This paper introduces AffinityNet, a CNN trained with image-level labels to predict pixel-level semantic affinities between adjacent coordinates, enabling random-walk-based propagation to refine CAMs and synthesize segmentation labels for training a strong segmentation model without extra annotations.

ABSTRACT

The deficiency of segmentation labels is one of the main obstacles to semantic segmentation in the wild. To alleviate this issue, we present a novel framework that generates segmentation labels of images given their image-level class labels. In this weakly supervised setting, trained models have been known to segment local discriminative parts rather than the entire object area. Our solution is to propagate such local responses to nearby areas which belong to the same semantic entity. To this end, we propose a Deep Neural Network (DNN) called AffinityNet that predicts semantic affinity between a pair of adjacent image coordinates. The semantic propagation is then realized by random walk with the affinities predicted by AffinityNet. More importantly, the supervision employed to train AffinityNet is given by the initial discriminative part segmentation, which is incomplete as a segmentation annotation but sufficient for learning semantic affinities within small image areas. Thus the entire framework relies only on image-level class labels and does not require any extra data or annotations. On the PASCAL VOC 2012 dataset, a DNN learned with segmentation labels generated by our method outperforms previous models trained with the same level of supervision, and is even as competitive as those relying on stronger supervision.

Motivation & Objective

• Address the lack of pixel-level annotations by leveraging image-level labels for segmentation.
• Learn pixel-level semantic affinities to propagate local discriminative responses to full object regions.
• Develop an end-to-end framework that synthesizes segmentation labels usable to train a segmentation model.
• Demonstrate state-of-the-art performance on PASCAL VOC 2012 under image-level supervision.

Proposed method

• Compute Class Activation Maps (CAMs) from an image-level trained classifier to seed object regions.
• Train AffinityNet to predict semantic affinity Wij between adjacent coordinates using a class-agnostic objective and CAM-derived supervision.
• Generate reliable pairwise affinity labels by selecting confident object/background regions from CAMs and dCRF refinements.
• Propagate CAMs via random walk using the learned affinity matrix to revise CAMs and obtain improved segmentation proposals.
• Upsample and refine revised CAMs with dCRF to synthesize segmentation labels for training a semantic segmentation network.
• Train the final segmentation model (e.g., Ours-ResNet38) on the synthesized labels.

Experimental results

Research questions

• RQ1 Can image-level labels be used to learn pixel-level semantic affinities that help recover full object shapes?
• RQ2 How effective is a learned affinity model in diffusing activation maps to accurate object boundaries via random walk?
• RQ3 Do synthesized segmentation labels enable competitive semantic segmentation under weak supervision?
• RQ4 How close can weakly supervised results come to fully supervised baselines on PASCAL VOC 2012?

Key findings

• AffinityNet trained with image-level supervision yields meaningful pixel-level affinities.
• Random walk with AffinityNet substantially improves CAM-based segmentation masks.
• Synthesized labels enable training a segmentation model that outperforms previous image-level supervised methods and is competitive with stronger supervision.
• Ours-ResNet38 achieves strong performance on PASCAL VOC 2012 compared to prior weakly supervised methods.
• The approach approaches the performance of fully supervised baselines, recovering a substantial portion of their performance.