[Paper Review] CANet: Class-Agnostic Segmentation Networks with Iterative Refinement and Attentive Few-Shot Learning

Chi Zhang, Guosheng Lin+3 more|arXiv (Cornell University)|Mar 6, 2019

Domain Adaptation and Few-Shot Learning28 references45 citations

TL;DR

CANet proposes a class-agnostic few-shot segmentation framework with a two-branch dense comparison module and an iterative refinement module, plus an attention-based k-shot fusion mechanism, achieving state-of-the-art mean IoU on PASCAL VOC 2012 for 1- and 5-shot settings and robustness with bounding-box supervision.

ABSTRACT

Recent progress in semantic segmentation is driven by deep Convolutional Neural Networks and large-scale labeled image datasets. However, data labeling for pixel-wise segmentation is tedious and costly. Moreover, a trained model can only make predictions within a set of pre-defined classes. In this paper, we present CANet, a class-agnostic segmentation network that performs few-shot segmentation on new classes with only a few annotated images available. Our network consists of a two-branch dense comparison module which performs multi-level feature comparison between the support image and the query image, and an iterative optimization module which iteratively refines the predicted results. Furthermore, we introduce an attention mechanism to effectively fuse information from multiple support examples under the setting of k-shot learning. Experiments on PASCAL VOC 2012 show that our method achieves a mean Intersection-over-Union score of 55.4% for 1-shot segmentation and 57.1% for 5-shot segmentation, outperforming state-of-the-art methods by a large margin of 14.6% and 13.2%, respectively.

Motivation & Objective

• Enable segmentation of unseen classes from few annotated examples (few-shot learning) in a class-agnostic setting.
• Leverage multi-level CNN features to perform dense pixel-wise comparisons between support and query images.
• Iteratively refine segmentation outputs to handle appearance variations within a class.
• Incorporate an attention mechanism to effectively fuse information from multiple support examples in k-shot scenarios.
• Explore bounding-box based weak supervision for support data to reduce labeling effort.

Proposed method

• Two-branch dense comparison module (DCM) that uses middle-level CNN features (block2 and block3 of ResNet-50 with dilations) to perform global foreground-guided dense comparison between support and query images.
• Global average pooling over the foreground region of the support image to obtain a global feature vector; concatenate and compare with every location in the query feature map; upsample and fuse via a 3x3 convolution block.
• Iterative optimization module (IOM) that refines predictions through residual connections that feed predicted masks into subsequent iterations; incorporates ASPP for multi-scale features.
• Training-time strategy includes end-to-end training with cross-entropy loss; uses a stochastic mask drop (p_r) to reduce overfitting of the IOM; initial forward pass without masks and optional incremental use of masks.
• Attention mechanism for k-shot segmentation that learns weights over k support examples to fuse their dense comparison features via a softmax-normalized attention score.

Experimental results

Research questions

• RQ1 Can a class-agnostic segmentation model generalize to unseen classes with only a few labeled examples?
• RQ2 Does a dense comparison approach using middle-level CNN features improve few-shot segmentation compared to prior methods?
• RQ3 Can iterative refinement enhance segmentation quality for few-shot tasks, particularly for complex object shapes?
• RQ4 Is learned attention-based fusion of multiple support examples more effective than non-learnable fusion methods in k-shot settings?
• RQ5 Is bounding-box supervision of the support set viable without substantial loss in segmentation performance?

Key findings

• CANet achieves mean IoU of 55.4% (1-shot) and 57.1% (5-shot) on PASCAL VOC 2012, outperforming prior methods by 14.6% and 13.2% respectively on meanIoU.
• The iterative optimization module provides a 2.8% improvement over the initial CANet predictions on PASCAL-5i.
• Attention-based fusion for k-shot markedly outperforms feature-averaging, mask-averaging, and logic-OR fusion in meanIoU on few-shot tasks.
• Bounding-box annotated support sets yield comparable performance to pixel-wise annotations (54.0% vs 52.0% meanIoU in Table 2 context), indicating label-efficiency in supervision.
• On COCO, 1-shot results improve with IOM by 4.1% meanIoU, and multi-scale evaluation adds a further gain; in 5-shot, attention-based fusion yields the best results among non-learnable baselines.