[Paper Review] Hypercorrelation Squeeze for Few-Shot Segmentation

Juhong Min, Dahyun Kang+1 more|arXiv (Cornell University)|Apr 4, 2021

Advanced Neural Network Applications81 references55 citations

TL;DR

HSNet introduces Hypercorrelation Squeeze Networks that fuse multi-level feature correlations and center-pivot 4D convolutions to achieve state-of-the-art few-shot semantic segmentation on standard benchmarks.

ABSTRACT

Few-shot semantic segmentation aims at learning to segment a target object from a query image using only a few annotated support images of the target class. This challenging task requires to understand diverse levels of visual cues and analyze fine-grained correspondence relations between the query and the support images. To address the problem, we propose Hypercorrelation Squeeze Networks (HSNet) that leverages multi-level feature correlation and efficient 4D convolutions. It extracts diverse features from different levels of intermediate convolutional layers and constructs a collection of 4D correlation tensors, i.e., hypercorrelations. Using efficient center-pivot 4D convolutions in a pyramidal architecture, the method gradually squeezes high-level semantic and low-level geometric cues of the hypercorrelation into precise segmentation masks in coarse-to-fine manner. The significant performance improvements on standard few-shot segmentation benchmarks of PASCAL-5i, COCO-20i, and FSS-1000 verify the efficacy of the proposed method.

Motivation & Objective

• Motivate and address the generalization gap in few-shot semantic segmentation with limited supervision.
• Leverage diverse mid- to high-level CNN features to build rich cross-image correspondences.
• Develop an efficient 4D convolutional framework to process dense hypercorrelations in a coarse-to-fine manner.
• Enable real-time inference through a center-pivot 4D kernel and a pyramidal encoder-decoder design.
• Extend the approach to K-shot settings with simple voting and masking mechanisms.

Proposed method

• Construct hypercorrelations by masking multi-level feature maps from support images and computing cosine-based 4D correlation tensors with query features.
• Concatenate correlated tensors across selected pyramid layers to form a hypercorrelation pyramid representing diverse semantic and geometric cues.
• Use a 4D-convolutional pyramid encoder with squeezing and mixing blocks to progressively condense hypercorrelations into a compact context feature Z.
• Decode the context feature with a 2D convolutional context decoder to predict foreground/background masks for the query image.
• Introduce center-pivot 4D convolution to reduce 4D kernel complexity via two 2D-convolutional components, enabling lighter and faster inferences.
• Extend to K-shot scenarios by running K forward passes and performing pixel-wise voting to obtain the final mask.

Experimental results

Research questions

• RQ1 How can diverse, multi-level feature correlations between support and query images be leveraged for accurate few-shot segmentation?
• RQ2 Can a lightweight, high-dimensional convolution (4D) architecture be made efficient enough for real-time inference while maintaining accuracy?
• RQ3 Does a coarse-to-fine, pyramid-based squeezing of hypercorrelations improve segmentation quality over single-level or single-scale approaches?
• RQ4 How does the proposed method perform under domain shift and with different backbone networks?
• RQ5 Can the approach be extended naturally to K-shot settings without loss of generality?

Key findings

• Proposes HSNet, achieving new state-of-the-art performance on PASCAL-5i, COCO-20i, and FSS-1000 benchmarks.
• Demonstrates that combining multi-level hypercorrelations with 4D convolutions yields substantial gains in 1-shot and 5-shot settings.
• Introduces center-pivot 4D convolution, providing a memory- and time-efficient alternative to standard 4D kernels while maintaining accuracy.
• Shows robustness to domain shift (COCO-trained models on PASCAL-5i) with competitive results and reduced parameters.
• Confirms that incorporating both semantic and geometric cues across a pyramid improves boundary refinement and localization, compared to single-layer correlations.