[Paper Review] Space-Time Correspondence as a Contrastive Random Walk

Allan Jabri, Andrew Owens+1 more|arXiv (Cornell University)|Jun 25, 2020

Human Pose and Action RecognitionComputer Science113 references116 citations

TL;DR

The paper introduces a self-supervised approach that learns visual space-time correspondence by casting it as a contrastive random walk on a video-derived space-time graph, guided by palindrome-based cycle-consistency and enhanced by edge dropout and test-time adaptation.

ABSTRACT

This paper proposes a simple self-supervised approach for learning a representation for visual correspondence from raw video. We cast correspondence as prediction of links in a space-time graph constructed from video. In this graph, the nodes are patches sampled from each frame, and nodes adjacent in time can share a directed edge. We learn a representation in which pairwise similarity defines transition probability of a random walk, so that long-range correspondence is computed as a walk along the graph. We optimize the representation to place high probability along paths of similarity. Targets for learning are formed without supervision, by cycle-consistency: the objective is to maximize the likelihood of returning to the initial node when walking along a graph constructed from a palindrome of frames. Thus, a single path-level constraint implicitly supervises chains of intermediate comparisons. When used as a similarity metric without adaptation, the learned representation outperforms the self-supervised state-of-the-art on label propagation tasks involving objects, semantic parts, and pose. Moreover, we demonstrate that a technique we call edge dropout, as well as self-supervised adaptation at test-time, further improve transfer for object-centric correspondence.

Motivation & Objective

• Learn a representation that captures visual correspondence across space and time from unlabeled video.
• Formulate correspondence as a path-finding problem on a space-time graph of video patches.
• Use cycle-consistency on palindrome sequences to provide supervision without labels.
• Improve robustness and transfer via edge dropout and test-time adaptation.

Proposed method

• Construct a directed space-time graph where nodes are patches from video frames and edges connect patches in neighboring frames based on learned similarities.
• Learn an embedding phi for patches so that pairwise similarities define a stochastic transition matrix for a random walk.
• Train using palindrome sequences to provide zero-shot targets, enforcing cycle-consistency across forward and backward walks.
• Formulate the learning as maximizing the likelihood of returning to the starting node along a path, equivalent to a contrastive learning objective.
• Introduce edge dropout on the transition matrix to encourage the walker to rely on alternative paths and improve grouping of common-fate regions.
• Optionally perform self-supervised adaptation at test-time by fine-tuning the embedding on unlabeled video prior to label propagation.

Experimental results

Research questions

• RQ1 Can a self-supervised representation learn robust visual correspondence from raw video data?
• RQ2 Can cycle-consistency with palindrome sequences provide supervision without ground-truth labels?
• RQ3 Does introducing edge dropout improve object-centric correspondence and segmentation tasks?
• RQ4 Does test-time self-supervision further enhance transfer to downstream label propagation tasks?

Key findings

• The learned representation, used as a similarity metric for label propagation, outperforms state-of-the-art self-supervised methods on tasks involving objects, pose keypoints, and semantic parts without task-specific adaptation.
• Increasing the length of walks during training improves downstream performance, indicating benefits of longer-range context.
• Edge dropout promotes robustness by forcing the model to rely on multiple plausible paths, improving object-centric correspondence.
• Test-time self-supervised adaptation yields further gains in object propagation quality, particularly in recall of segmentation quality.
• The approach scales with longer walks and can be extended through simple extensions without complex supervision.