[Paper Review] Universal Correspondence Network

Christopher Choy, JunYoung Gwak+2 more|arXiv (Cornell University)|Jun 11, 2016

Advanced Image and Video Retrieval TechniquesComputer Science25 references257 citations

TL;DR

The paper presents a deep metric learning framework (UCN) for generic visual correspondences, enabling geometric and semantic matching with a fully convolutional architecture, a novel correspondence contrastive loss, and a convolutional spatial transformer for patch normalization.

ABSTRACT

We present a deep learning framework for accurate visual correspondences and demonstrate its effectiveness for both geometric and semantic matching, spanning across rigid motions to intra-class shape or appearance variations. In contrast to previous CNN-based approaches that optimize a surrogate patch similarity objective, we use deep metric learning to directly learn a feature space that preserves either geometric or semantic similarity. Our fully convolutional architecture, along with a novel correspondence contrastive loss allows faster training by effective reuse of computations, accurate gradient computation through the use of thousands of examples per image pair and faster testing with $O(n)$ feed forward passes for $n$ keypoints, instead of $O(n^2)$ for typical patch similarity methods. We propose a convolutional spatial transformer to mimic patch normalization in traditional features like SIFT, which is shown to dramatically boost accuracy for semantic correspondences across intra-class shape variations. Extensive experiments on KITTI, PASCAL, and CUB-2011 datasets demonstrate the significant advantages of our features over prior works that use either hand-constructed or learned features.

Motivation & Objective

• Motivate learning a feature space where geometric and semantic similarity is preserved directly for visual correspondences.
• Develop a fully convolutional network to enable dense, scalable feature extraction and efficient testing.
• Introduce a correspondence contrastive loss to efficiently train with thousands of correspondences per image pair.
• Propose a convolutional spatial transformer to mimic patch normalization and improve invariance to intra-class variations.
• Demonstrate state-of-the-art performance on geometric and semantic matching benchmarks (KITTI, PASCAL, CUB).

Proposed method

• Train a fully convolutional network to learn a metric space where corresponding points have nearby features and non-corresponding points are separated by a margin m.
• Use a correspondence contrastive loss that scales with thousands of correspondences per image pair, enabling O(n) test-time passes instead of O(n^2).
• Incorporate on-the-fly hard negative mining to focus training on the most informative negatives.
• Add a convolutional spatial transformer to apply independent, per-keypoint spatial transformations for patch normalization and invariance to affine distortions.
• Extract dense features densely, with channel-wise L2 normalization, and perform nearest-neighbor matching in feature space at test time.
• Optional Siamese/contrastive variants and ablations (hard negative mining, spatial transformer) to assess component contributions.

Experimental results

Research questions

• RQ1 Can a learned metric space directly optimize for geometric and semantic correspondences across rigid and non-rigid variations?
• RQ2 Does a fully convolutional architecture with dense feature extraction enable faster training and testing for correspondence tasks?
• RQ3 Can a correspondence-specific loss and hard-negative mining improve correspondence accuracy over patch-similarity-based approaches?
• RQ4 Does a convolutional spatial transformer improve robustness to intra-class shape variations in semantic matching?
• RQ5 How does UCN perform on standard geometric and semantic correspondence benchmarks compared with hand-crafted and prior learned methods?

Key findings

• UCN achieves dense and accurate correspondences for both geometric and semantic tasks, outperforming prior methods on several benchmarks.
• On KITTI flow and MPI-Sintel, UCN variants with hard negative mining and spatial transformer reach top performance, e.g., MPI-Sintel: Ours-HN 91.5 and Ours-HN-ST 90.7 (PCK measures).
• KITTI results show strong performance with Ours-HN 86.5 and Ours-HN-ST 83.4 in Table 3, surpassing several traditional and CNN-based baselines.
• Semantic correspondence on PASCAL-Berkeley and CUB datasets benefits significantly from the convolutional spatial transformer, with notable gains over prior methods.
• Camera motion estimation on KITTI raw sequences using UCN features yields competitive essential matrix decomposition results compared to sparse baselines.
• The proposed method yields superior performance without relying on post-processing like global MRF optimization, highlighting the strength of the learned metric space and dense features.