[Paper Review] CRNet: Cross-Reference Networks for Few-Shot Segmentation

Weide Liu, Chi Zhang+2 more|arXiv (Cornell University)|Mar 24, 2020

Domain Adaptation and Few-Shot Learning27 references19 citations

TL;DR

CRNet proposes a symmetric cross-reference network for few-shot image segmentation that jointly predicts masks in both support and query images, leveraging co-occurrent features through a cross-reference module and improving predictions via a recurrent mask refinement module with confidence caching. The method achieves state-of-the-art performance on PASCAL VOC 2012, especially in 5-shot settings, outperforming prior methods by up to 8.4 mIoU when finetuning on multiple support images.

ABSTRACT

Over the past few years, state-of-the-art image segmentation algorithms are based on deep convolutional neural networks. To render a deep network with the ability to understand a concept, humans need to collect a large amount of pixel-level annotated data to train the models, which is time-consuming and tedious. Recently, few-shot segmentation is proposed to solve this problem. Few-shot segmentation aims to learn a segmentation model that can be generalized to novel classes with only a few training images. In this paper, we propose a cross-reference network (CRNet) for few-shot segmentation. Unlike previous works which only predict the mask in the query image, our proposed model concurrently make predictions for both the support image and the query image. With a cross-reference mechanism, our network can better find the co-occurrent objects in the two images, thus helping the few-shot segmentation task. We also develop a mask refinement module to recurrently refine the prediction of the foreground regions. For the $k$-shot learning, we propose to finetune parts of networks to take advantage of multiple labeled support images. Experiments on the PASCAL VOC 2012 dataset show that our network achieves state-of-the-art performance.

Motivation & Objective

• To address the challenge of few-shot image segmentation, where models must generalize to novel categories with only a few annotated examples.
• To overcome the limitation of unidirectional guidance in prior methods, which only use support images to guide query image segmentation.
• To improve mask prediction accuracy by enabling mutual refinement between support and query images through a symmetric architecture.
• To develop an effective mask refinement mechanism that iteratively improves predictions using a confidence cache.
• To propose a finetuning-based strategy for k-shot learning that scales performance with increasing support image count, unlike fusion-based baselines that saturate.

Proposed method

• The CRNet employs a two-branch encoder-decoder architecture that simultaneously predicts foreground masks in both the support and query images.
• A cross-reference module compares features between the two images to generate reinforced representations by identifying co-occurrent objects, enhancing feature discrimination.
• The mask refinement module uses a confidence map as a cache, iteratively refining predictions by updating the cache with new probability maps at each step.
• For k-shot learning, the network is finetuned using up to k² image pairs derived from k support images, enabling better utilization of multiple labeled examples.
• The network is trained with a combined loss: a main segmentation loss on the query image and an auxiliary loss from the co-occurrence prediction task in the cross-reference module.
• Multi-scale inference and multi-level feature fusion are applied during testing to improve robustness to scale variations and enhance localization accuracy.

Experimental results

Research questions

• RQ1 Can symmetric prediction in both support and query images improve few-shot segmentation performance compared to unidirectional guidance?
• RQ2 How effective is a confidence cache-based mask refinement module in iteratively improving foreground mask predictions?
• RQ3 Does finetuning the network on multiple support images lead to better performance than feature or mask fusion in k-shot segmentation?
• RQ4 How do multi-level features and multi-scale testing affect the robustness and accuracy of the model?
• RQ5 What is the contribution of the cross-reference module in improving feature representation for co-occurrent object detection and segmentation?

Key findings

• The cross-reference module alone improves performance by over 10% mIoU compared to a baseline without it, demonstrating its critical role in feature enhancement.
• The mask refinement module with confidence caching increases mIoU by 3.1 points over a single-inference baseline, showing significant improvement in prediction quality.
• In the 5-shot setting, the finetuning-based method outperforms the fusion-based baseline by 8.4 mIoU, and continues improving with more support images, while the fusion method begins to degrade.
• Multi-scale testing improves mIoU by 1.2 points in the 1-shot setting, indicating robustness to scale variations.
• Using multi-level features instead of only the final layer features improves mIoU by 1.8 points, highlighting the importance of mid-level contextual features.
• On MS COCO, ablation studies confirm that both the cross-reference and mask refinement modules contribute significantly to performance, with the full model achieving the highest mIoU.