[Paper Review] Self-supervised Learning for Video Correspondence Flow

Zihang Lai, Weidi Xie|arXiv (Cornell University)|May 2, 2019

Video Surveillance and Tracking Methods52 references56 citations

TL;DR

The paper trains RGB video embeddings for pixelwise correspondence flow using self-supervised frame reconstruction with an information bottleneck,Restricted Attention, scheduled sampling, and cycle consistency, achieving state-of-the-art on DAVIS-2017 and JHMDB without fine-tuning.

ABSTRACT

The objective of this paper is self-supervised learning of feature embeddings that are suitable for matching correspondences along the videos, which we term correspondence flow. By leveraging the natural spatial-temporal coherence in videos, we propose to train a ``pointer'' that reconstructs a target frame by copying pixels from a reference frame. We make the following contributions: First, we introduce a simple information bottleneck that forces the model to learn robust features for correspondence matching, and prevent it from learning trivial solutions, \eg matching based on low-level colour information. Second, to tackle the challenges from tracker drifting, due to complex object deformations, illumination changes and occlusions, we propose to train a recursive model over long temporal windows with scheduled sampling and cycle consistency. Third, we achieve state-of-the-art performance on DAVIS 2017 video segmentation and JHMDB keypoint tracking tasks, outperforming all previous self-supervised learning approaches by a significant margin. Fourth, in order to shed light on the potential of self-supervised learning on the task of video correspondence flow, we probe the upper bound by training on additional data, \ie more diverse videos, further demonstrating significant improvements on video segmentation.

Motivation & Objective

• Motivate learning robust, pixel-level correspondences in videos without manual annotations.
• Leverage temporal-spatial coherence in videos to train embeddings via frame reconstruction.
• Prevent trivial solutions and tracker drift through architectural and training strategies.
• Demonstrate strong performance on downstream tasks like video segmentation and keypoint tracking.
• Explore upper bounds by training on more diverse, unlabeled video data.

Proposed method

• Use a ResNet-18 based encoder to produce embeddings with an information bottleneck that perturbs RGB channels and intensities.
• Implement restricted attention to compute dense correspondence within a local search window (disparity M).
• Train a recursive, long-range model over short sequences with scheduled sampling to bridge train/test gaps.
• Apply forward-backward cycle consistency as a regularizer against drift in propagation.
• Frame reconstruction is cast as a color quantization classification task with 16 color classes (K-means in Lab space) and cross-entropy loss.
• Train entirely self-supervised on Kinetics, decoding at 6 fps, with RGB input and optional color dropout to encourage robustness.

Experimental results

Research questions

• RQ1 Can self-supervised learning yield embeddings suitable for dense video correspondence flow without labeled data?
• RQ2 Does frame reconstruction with an information bottleneck prevent reliance on low-level color cues and reduce trivial solutions?
• RQ3 Will scheduled sampling and cycle consistency mitigate tracker drift over long sequences?
• RQ4 How does restricted attention affect performance and scalability to higher-resolution inputs?
• RQ5 What is the upper bound of performance when training on more diverse unlabeled videos?

Key findings

• Achieves state-of-the-art among self-supervised methods on DAVIS-2017 video segmentation (J&F mean 49.5/47.7? see table) and JHMDB keypoint tracking (PCK metrics).
• Color dropout and RGB input are crucial to bridge train/test discrepancy and markedly improve DAVIS-2017 metrics.
• Restricted attention reduces memory by ~1.4G GPU usage and, while causing some drop, still yields competitive performance compared to full attention.
• Scheduled sampling significantly improves robustness and reduces drift, especially under illumination changes.
• Forward-backward cycle consistency markedly improves long-sequence propagation, reducing drift and boosting J&F scores.
• Upper-bound experiments show additional unlabeled video data further boosts performance, approaching or surpassing some supervised baselines for several classes.