[Paper Review] Few-Shot Segmentation Propagation with Guided Networks

Kate Rakelly, Evan Shelhamer+3 more|arXiv (Cornell University)|May 25, 2018

Domain Adaptation and Few-Shot LearningComputer Science26 references103 citations

TL;DR

This paper introduces guided networks for few-shot segmentation that produce a task representation from sparse support annotations and segment new inputs without further optimization, enabling rapid propagation across images, time, and scenes.

ABSTRACT

Learning-based methods for visual segmentation have made progress on particular types of segmentation tasks, but are limited by the necessary supervision, the narrow definitions of fixed tasks, and the lack of control during inference for correcting errors. To remedy the rigidity and annotation burden of standard approaches, we address the problem of few-shot segmentation: given few image and few pixel supervision, segment any images accordingly. We propose guided networks, which extract a latent task representation from any amount of supervision, and optimize our architecture end-to-end for fast, accurate few-shot segmentation. Our method can switch tasks without further optimization and quickly update when given more guidance. We report the first results for segmentation from one pixel per concept and show real-time interactive video segmentation. Our unified approach propagates pixel annotations across space for interactive segmentation, across time for video segmentation, and across scenes for semantic segmentation. Our guided segmentor is state-of-the-art in accuracy for the amount of annotation and time. See http://github.com/shelhamer/revolver for code, models, and more details.

Motivation & Objective

• Reduce annotation burden by enabling segmentation with very few pixel-level annotations.
• Develop a task-driven, end-to-end trainable framework that can switch tasks with guidance.
• Propagate annotations across space (images), time (video), and scenes (semantic classes) using a unified approach.
• Enable real-time interactive updates when new guidance becomes available.

Proposed method

• Introduce guided networks with a guide g that extracts a latent task representation z from sparse annotations.
• Use a two-branch architecture: a guidance (g) branch that summarizes the support into z, and an inference (f) branch that segments the query given z.
• Adopt late fusion of image and annotation streams to form z, enabling fast updates by recomputing masks rather than features.
• Global pooling of per-location task representations to form a task vector that can propagate across space and time.
• Train with episodic, few-shot segmentation tasks synthesized from dense segmentation data, optimizing both the guide and the segmentor end-to-end.
• Evaluate conditioning strategies (feature fusion, parameter regression, prototypes) and select feature fusion as the preferred guiding mechanism.

Experimental results

Research questions

• RQ1 Can a single, feedforward guided network handle extremely sparse to dense annotations across multiple segmentation tasks?
• RQ2 How should support information (annotations) be fused with images to form an effective task representation for guided inference?
• RQ3 What is the best guidance strategy for structured output segmentation (feature fusion vs. parameter regression vs. prototypes) in a few-shot setting?
• RQ4 How does the proposed method perform on interactive image segmentation, semantic segmentation, and video object segmentation, including real-time interactive video segmentation?
• RQ5 Does the model maintain accuracy as the amount of guidance (S shots, P pixels) varies, and can it update quickly with additional annotations?

Key findings

• The guided network achieves state-of-the-art accuracy for sparse annotations across interactive segmentation, video object segmentation, and few-shot semantic segmentation.
• Late fusion of image and annotation streams with global pooling improves accuracy and enables rapid updates when new annotations are provided.
• For interactive and video tasks, guidance enables real-time or near-real-time updates, significantly faster than fine-tuning approaches.
• In semantic segmentation on Pascal VOC, the method attains strong performance with as few as two annotated pixels per class, rivaling dense-annotation baselines in sparse regimes.
• On DAVIS 2017 video object segmentation, the method is state-of-the-art in the sparse regime and shows substantial dense-regime accuracy with competitive speed (e.g., 33.3% accuracy in the dense regime with 80% relative improvement over peers in the same time envelope).
• The approach supports cross-mode transfer, performing across image, video, and semantic tasks with a single model trained on a single mode.