[Paper Review] Learning Two-View Correspondences and Geometry Using Order-Aware Network

Jiahui Zhang, Dawei Sun+7 more|arXiv (Cornell University)|Aug 14, 2019

Advanced Vision and Imaging45 references54 citations

TL;DR

The paper introduces Order-Aware Network (OANet) to jointly predict inlier probabilities of sparse correspondences and regress the essential matrix, using DiffPool, Order-Aware DiffUnpool, and an Order-Aware Filtering block to capture local and global context for improved two-view geometry estimation.

ABSTRACT

Establishing correspondences between two images requires both local and global spatial context. Given putative correspondences of feature points in two views, in this paper, we propose Order-Aware Network, which infers the probabilities of correspondences being inliers and regresses the relative pose encoded by the essential matrix. Specifically, this proposed network is built hierarchically and comprises three novel operations. First, to capture the local context of sparse correspondences, the network clusters unordered input correspondences by learning a soft assignment matrix. These clusters are in a canonical order and invariant to input permutations. Next, the clusters are spatially correlated to form the global context of correspondences. After that, the context-encoded clusters are recovered back to the original size through a proposed upsampling operator. We intensively experiment on both outdoor and indoor datasets. The accuracy of the two-view geometry and correspondences are significantly improved over the state-of-the-arts. Code will be available at https://github.com/zjhthu/OANet.git.

Motivation & Objective

• Motivate accurate two-view geometry estimation by leveraging both local and global context for sparse correspondences.
• Develop permutation-invariant clustering of correspondences to capture local context without predefined neighbors.
• Enable hierarchical context modeling and precise upsampling to original correspondences.
• Recover essential matrix end-to-end via a differentiable, weighted eight-point algorithm.
• Demonstrate improved pose estimation on outdoor and indoor datasets compared to prior methods.

Proposed method

• Introduce Differentiable Pooling (DiffPool) to cluster unordered correspondences into a canonical order.
• Develop Order-Aware Differentiable Unpooling (DiffUnpool) to upsample clusters back to original correspondences with order awareness.
• Add Order-Aware Filtering block to model global context by correlating clusters in canonical order.
• Use a permutation-equivariant f_phi to produce per-correspondence logits z and weights w for inlier likelihood.
• Regress the essential matrix E_hat via a differentiable weighted eight-point algorithm g(w, C).
• Train with a combination of inlier classification loss and an essential matrix loss (L2 or geometry loss).

Experimental results

Research questions

• RQ1 Can a permutation-invariant clustering of sparse correspondences capture local context to improve outlier rejection?
• RQ2 Does a hierarchy of learned pooling/unpooling and order-aware filtering better model global context for two-view geometry?
• RQ3 Can end-to-end differentiable weighting of correspondences improve essential matrix regression beyond traditional robust methods?
• RQ4 What is the impact of the proposed components (DiffPool, DiffUnpool, Order-Aware Filtering) on accuracy of relative pose under outdoor and indoor conditions?

Key findings

• The proposed DiffPool and Order-Aware DiffUnpool enable learning to cluster and upsample correspondences with a canonical order, improving local context modeling.
• Order-Aware Filtering blocks significantly improve global context modeling over baseline PointCN blocks in the second level of the network.
• Ablation studies show that Order-Aware DiffUnpool yields notable gains (e.g., 5.23% improvement over the baseline on unknown scenes without RANSAC).
• Replacing PointCN blocks with Order-Aware Filtering blocks yields further gains, especially on unknown scenes without RANSAC.
• An iterative network variant and geometry loss further bolster performance, with consistent improvements on outdoor (YFCC100M) and indoor (SUN3D) datasets.
• Compared to baselines (PointCN, PointNet++, N3Net, DFE), the proposed method achieves the best mAP under various settings, including unknown scenes and with/without RANSAC.