[Paper Review] FusionSeg: Learning to combine motion and appearance for fully automatic segmention of generic objects in videos

Suyog Dutt Jain, Bo Xiong+1 more|arXiv (Cornell University)|Jan 19, 2017

Visual Attention and Saliency Detection44 references151 citations

TL;DR

FusionSeg presents a two-stream CNN that jointly learns appearance and motion cues to produce pixel-level foreground segmentation for generic objects in videos, trained with weak video data and image annotations; it achieves state-of-the-art results on DAVIS, YouTube-Objects, and SegTrack-v2.

ABSTRACT

We propose an end-to-end learning framework for segmenting generic objects in videos. Our method learns to combine appearance and motion information to produce pixel level segmentation masks for all prominent objects in videos. We formulate this task as a structured prediction problem and design a two-stream fully convolutional neural network which fuses together motion and appearance in a unified framework. Since large-scale video datasets with pixel level segmentations are problematic, we show how to bootstrap weakly annotated videos together with existing image recognition datasets for training. Through experiments on three challenging video segmentation benchmarks, our method substantially improves the state-of-the-art for segmenting generic (unseen) objects. Code and pre-trained models are available on the project website.

Motivation & Objective

• Motivate and enable fully automatic segmentation of generic objects in videos without relying on category-specific models.
• Leverage complementary cues from appearance and motion to improve pixel-level segmentation.
• Develop a training strategy that bootstraps from image annotations and weak video data due to lack of large pixel-level video datasets.
• Demonstrate state-of-the-art performance on multiple challenging video segmentation benchmarks.

Proposed method

• Two-stream fully convolutional network that processes RGB frames (appearance stream) and color-coded optical flow (motion stream).
• Appearance stream builds on a dilated-ResNet-101 with multi-scale parallel dilated branches to produce per-pixel objectness maps.
• Motion stream uses the same architecture but trained on optical flow inputs, bootstrapped using pseudo-ground truth derived from bounding boxes and flow-based filtering.
• Fusion model combines the streams via three branches: appearance then motion, motion then appearance, and a pairwise product, followed by a max operation to yield final segmentation."
• Training data is gathered by bootstrapping from image segmentation datasets (e.g., PASCAL VOC 2012) and weak bounding-box video annotations (ImageNet-Video), with filtering steps to generate high-quality pseudo-ground truth for motion network training.

Experimental results

Research questions

• RQ1 Can appearance and motion cues be fused in an end-to-end trainable framework to segment generic foreground objects in videos automatically?
• RQ2 How can a deep model be trained for video segmentation when large-scale pixel-level video annotations are unavailable?
• RQ3 Does jointly modeling appearance and motion outperform using either cue alone on diverse video benchmarks?
• RQ4 What training strategy can leverage image segmentation data and weak video annotations to learn robust pixel-level segmentation for moving and static objects?

Key findings

• The joint appearance+motion model yields significant improvements over individual streams (up to 11 percentage points on DAVIS).
• On DAVIS, the joint model (Ours-Joint) achieves 71.51% average IoU, outperforming several automatic and some semi-supervised baselines.
• On YouTube-Objects, the joint model achieves 68.43% average IoU, outperforming many state-of-the-art automatic methods and remaining competitive with semi-supervised approaches.
• On SegTrack-v2, the joint model achieves 61.40% average IoU, surpassing several baselines and demonstrating the benefit of fusing motion and appearance across datasets.
• The approach outperforms fully automatic methods and even some semi-supervised methods that require human input, while operating fully automatically at test time.