[論文レビュー] Efficient Video Object Segmentation via Network Modulation

Linjie Yang, Yanran Wang+3 more|arXiv (Cornell University)|Feb 4, 2018

Visual Attention and Saliency Detection参考文献 27被引用数 40

ひとこと要約

One or two sentence direct-answer summary

ABSTRACT

Video object segmentation targets at segmenting a specific object throughout a video sequence, given only an annotated first frame. Recent deep learning based approaches find it effective by fine-tuning a general-purpose segmentation model on the annotated frame using hundreds of iterations of gradient descent. Despite the high accuracy these methods achieve, the fine-tuning process is inefficient and fail to meet the requirements of real world applications. We propose a novel approach that uses a single forward pass to adapt the segmentation model to the appearance of a specific object. Specifically, a second meta neural network named modulator is learned to manipulate the intermediate layers of the segmentation network given limited visual and spatial information of the target object. The experiments show that our approach is 70times faster than fine-tuning approaches while achieving similar accuracy.

研究の動機と目的

• Motivate and address the inefficiency of online fine-tuning for semi-supervised video object segmentation in one-shot settings.
• Develop a meta-learner (modulator) that instantly adapts a base segmentation network to a specific object using limited first-frame cues.
• Incorporate both visual appearance and spatial priors to guide network modulation for robust tracking across frames.
• Demonstrate that modulation-based adaptation achieves competitive accuracy with substantial speed improvements over fine-tuning approaches.

提案手法

• Introduce two modulators: a visual modulator that outputs channel-wise scale parameters for modulation layers, and a spatial modulator that outputs pixel-wise biases using a spatial prior heatmap.
• Use a modulation layer after most convolutional layers where y_c = gamma_c * x_c + beta_c, with gamma from visual modulator and beta from spatial modulator.
• Visual modulator processes the annotated object image (visual guide) via a modified VGG16 to produce modulation parameters.
• Spatial modulator takes a prior location (previous frame mask) encoded as a Gaussian heatmap, downsamples it to match feature map resolutions, and generates biases.
• Train the system end-to-end with a two-input setup (visual + spatial cues) on MS-COCO and finetune on video data if desired; use a balanced cross-entropy loss.
• Maintain a fully-convolutional main segmentation network (VGG16-based with hyper-column features) with modulation layers after all convolutional layers except the first four.

実験結果

リサーチクエスチョン

• RQ1 Can a secondary meta-network learn to instantly adapt a segmentation model to a specific object without iterative fine-tuning?
• RQ2 Does combining visual appearance guidance with a spatial prior improve robustness to multiple similar objects and object motion?
• RQ3 What is the performance-speed trade-off of network modulation compared to traditional online fine-tuning in semi-supervised video segmentation?
• RQ4 How well do modulation parameters correlate with object appearance and trackability across frames?

主な発見

方法	DAVIS 16 (平均 IU)	YoutubeObjs (平均 IU)	with FT	OptFlow	CRF	速度 (s)
Ours (Stage 1)	72.2	66.4	✗	✗	✗	0.14
Ours (Stage 1&2)	74.0	69.0	✗	✗	✗	0.14
Ours	52.5	60.9	✗	✗	✗	-

• The proposed network modulation approach achieves about 70x speedup compared to online fine-tuning while attaining similar accuracy.
• On DAVIS 2016 and YoutubeObjects, the method outperforms non-finetuning baselines and is competitive with finetuned methods.
• DAVIS 2017 results show substantial gains over MaskTrack-B and OSVOS-B without finetuning, and gains when using modulation on finetuned baselines.
• Visualization reveals that modulation parameters form meaningful embeddings for object categories, with deeper layers showing larger parameter variation.
• The spatial prior biases are sparse in early layers and become more pronounced in deeper layers, indicating gradual integration of spatial cues into features.