[Paper Review] AffineGlue: Joint Matching and Robust Estimation

Dániel Baráth, Dmytro Mishkin+4 more|arXiv (Cornell University)|Jul 28, 2023

Face and Expression Recognition4 citations

TL;DR

AffineGlue proposes a joint feature matching and robust estimation framework that leverages one-to-many correspondences and single-point minimal solvers to reduce combinatorial complexity and improve robustness. By using a novel 1-affine correspondence (1AC) solver with gravity prior and a neural rejection head, it achieves state-of-the-art performance, improving AUC@10° by 6.6 points on PhotoTourism and enabling SuperPoint+SuperGlue to match LoFTR's accuracy on ScanNet.

ABSTRACT

We propose AffineGlue, a method for joint two-view feature matching and robust estimation that reduces the combinatorial complexity of the problem by employing single-point minimal solvers. AffineGlue selects potential matches from one-to-many correspondences to estimate minimal models. Guided matching is then used to find matches consistent with the model, suffering less from the ambiguities of one-to-one matches. Moreover, we derive a new minimal solver for homography estimation, requiring only a single affine correspondence (AC) and a gravity prior. Furthermore, we train a neural network to reject ACs that are unlikely to lead to a good model. AffineGlue is superior to the SOTA on real-world datasets, even when assuming that the gravity direction points downwards. On PhotoTourism, the AUC@10° score is improved by 6.6 points compared to the SOTA. On ScanNet, AffineGlue makes SuperPoint and SuperGlue achieve similar accuracy as the detector-free LoFTR.

Motivation & Objective

• To address the instability of traditional one-to-one matching in low-inlier-ratio scenarios, especially under repeated patterns and wide baselines.
• To reduce the combinatorial complexity of robust estimation by replacing multi-point minimal solvers with single-point solvers based on affine correspondences.
• To improve robustness and accuracy in relative pose and homography estimation by integrating a gravity prior and neural rejection of poor affine correspondences.
• To enable state-of-the-art performance with standard feature detectors and matchers like SuperPoint+SuperGlue without requiring detector-free architectures.
• To demonstrate that joint matching and estimation via guided sampling from one-to-many matches outperforms conventional RANSAC-based pipelines.

Proposed method

• AffineGlue constructs one-to-many correspondences from feature detectors like SuperPoint and descriptors like SuperGlue, reducing ambiguity in matching.
• It employs a novel minimal solver that estimates homography using only one affine correspondence (1AC) and a gravity prior, significantly reducing solver complexity.
• A neural network is trained to reject low-quality affine correspondences before model estimation, improving robustness and convergence.
• Guided sampling iteratively selects inlier-consistent one-to-one matches based on the estimated model, enabling accurate scoring and refinement.
• The method uses iterative hypothesis generation and verification via a single-point solver, avoiding the need for multi-point minimal solvers.
• It integrates a gravity prior (assumed downward) to stabilize the 1AC solver, even when the actual gravity direction is inaccurate (e.g., 24.8° error on ScanNet).

Experimental results

Research questions

• RQ1 Can one-to-many correspondences improve robustness in feature matching under ambiguous or repeated patterns?
• RQ2 Can a single-point minimal solver based on affine correspondences and gravity prior outperform traditional multi-point solvers in homography estimation?
• RQ3 Does joint matching and robust estimation via guided sampling lead to higher accuracy than sequential matching and RANSAC?
• RQ4 Can a learned rejection head for affine correspondences improve model quality and reduce runtime?
• RQ5 To what extent does the gravity prior affect performance when it is only a rough approximation in real-world data?

Key findings

• On PhotoTourism, AffineGlue improves AUC@10° by 6.6 points compared to the previous SOTA, achieving 58.1% AUC.
• On ScanNet, AffineGlue enables SuperPoint+SuperGlue to match the accuracy of detector-free LoFTR, achieving 89.2% AUC@20°.
• The 1AC+gravity solver with neural rejection achieves a 5-point AUC@1° improvement over the second-best method on HPatches.
• AffineGlue runs in real-time, with average inference times of 0.04s for homography and 0.09s for pose estimation on PhotoTourism.
• SuperPoint with SuperGlue and AffNet outperforms DoG+HardNet by 1.5–2.5 AUC points on PhotoTourism, even though HardNet is not trained on affine features.
• The method remains robust even with a gravity prior error of 24.8° on ScanNet, demonstrating strong tolerance to imperfect prior knowledge.