[Paper Review] Blazingly Fast Video Object Segmentation with Pixel-Wise Metric Learning

Yuhua Chen, Jordi Pont-Tuset+2 more|arXiv (Cornell University)|Apr 9, 2018

Visual Attention and Saliency DetectionComputer Science38 references17 citations

TL;DR

This paper proposes a blazingly fast video object segmentation method using pixel-wise metric learning via a fully convolutional network trained with a modified triplet loss. By embedding pixels of the same object close in a learned space and classifying others via nearest-neighbor search, it achieves state-of-the-art accuracy at 275ms per frame in semi-supervised settings and reaches comparable performance with only 0.15 clicks per frame in interactive scenarios.

ABSTRACT

This paper tackles the problem of video object segmentation, given some user annotation which indicates the object of interest. The problem is formulated as pixel-wise retrieval in a learned embedding space: we embed pixels of the same object instance into the vicinity of each other, using a fully convolutional network trained by a modified triplet loss as the embedding model. Then the annotated pixels are set as reference and the rest of the pixels are classified using a nearest-neighbor approach. The proposed method supports different kinds of user input such as segmentation mask in the first frame (semi-supervised scenario), or a sparse set of clicked points (interactive scenario). In the semi-supervised scenario, we achieve results competitive with the state of the art but at a fraction of computation cost (275 milliseconds per frame). In the interactive scenario where the user is able to refine their input iteratively, the proposed method provides instant response to each input, and reaches comparable quality to competing methods with much less interaction.

Motivation & Objective

• Address the lack of real-time, accurate video object segmentation for interactive and real-world applications.
• Overcome the high computational cost of existing state-of-the-art methods, which often require seconds per frame.
• Enable efficient, interactive segmentation where users can iteratively refine results with minimal input.
• Develop a unified framework that supports diverse user inputs such as clicks, scribbles, or masks without retraining.
• Achieve high accuracy with minimal user interaction by leveraging a learned embedding space for pixel-wise retrieval.

Proposed method

• Train a fully convolutional network (FCN) to map video pixels into a metric embedding space using a modified triplet loss tailored for video object segmentation.
• Formulate video object segmentation as pixel-wise retrieval: annotated pixels serve as reference points, and all other pixels are classified via nearest-neighbor search in the embedding space.
• Decouple the embedding computation from user input, allowing pre-computed embeddings to be reused across multiple user interactions.
• Support diverse user inputs (e.g., clicks, masks, scribbles) within a single unified inference pipeline.
• Use a robot-simulated user study to evaluate interactive performance, with iterative refinement based on misclassified pixels.
• Extend the method to multiple-object scenarios by treating each object as a separate reference cluster in the embedding space.

Experimental results

Research questions

• RQ1 Can a metric learning-based approach achieve state-of-the-art accuracy in video object segmentation while maintaining sub-second inference speed?
• RQ2 How effective is pixel-wise retrieval in a learned embedding space for both semi-supervised and interactive video segmentation?
• RQ3 To what extent can user interaction be minimized without sacrificing segmentation quality?
• RQ4 How robust is the method to the choice of user-provided clicks, especially in the absence of ground-truth guidance?
• RQ5 Can the method generalize effectively to multiple-object videos with occlusions and complex interactions?

Key findings

• The method achieves 77.5% J&F on DAVIS 2016 in the semi-supervised setting with only 275ms per frame, outperforming other fast methods in speed-accuracy trade-off.
• In the interactive scenario, the method reaches 74.5% J&F with just 10 clicks across an entire video, demonstrating high efficiency in user interaction.
• With 0.15 clicks per frame, the method achieves a J score of 75.5%, matching the performance of full-mask first-frame annotation but with drastically reduced user effort.
• Real-user studies show an average of 0.17 clicks per frame are sufficient to achieve 77.7% J score, completing segmentation in about 24 seconds versus over 3 minutes for full-mask annotation.
• The method generalizes well to multiple-object videos, achieving competitive results on DAVIS 2017 with only 0.5 clicks per frame, significantly outperforming baseline methods in efficiency.
• The variance in performance across repeated simulations is low (0.1 at 1 click per frame), indicating robustness to the selection of click points.