[Paper Review] Visual Correspondence Hallucination

Hugo Germain, Vincent Lepetit+1 more|arXiv (Cornell University)|Jun 17, 2021

Advanced Vision and Imaging5 citations

TL;DR

This paper introduces NeurHal, a deep learning method that hallucinates visual correspondences—predicting keypoint matches in target images even when they are occluded or outside the field of view. By training a network to output peaked probability distributions over correspondent locations, NeurHal generalizes to unseen scenes and significantly improves robustness in absolute camera pose estimation compared to state-of-the-art local feature matching methods.

ABSTRACT

Given a pair of partially overlapping source and target images and a keypoint in the source image, the keypoint's correspondent in the target image can be either visible, occluded or outside the field of view. Local feature matching methods are only able to identify the correspondent's location when it is visible, while humans can also hallucinate its location when it is occluded or outside the field of view through geometric reasoning. In this paper, we bridge this gap by training a network to output a peaked probability distribution over the correspondent's location, regardless of this correspondent being visible, occluded, or outside the field of view. We experimentally demonstrate that this network is indeed able to hallucinate correspondences on pairs of images captured in scenes that were not seen at training-time. We also apply this network to an absolute camera pose estimation problem and find it is significantly more robust than state-of-the-art local feature matching-based competitors.

Motivation & Objective

• To investigate whether a neural network can be trained to learn and predict visual correspondences even when they are not directly visible in the target image.
• To bridge the gap between human-like geometric reasoning and current local feature matching methods, which fail on non-covisible regions.
• To evaluate whether correspondence hallucination enhances performance in absolute camera pose estimation.
• To design a novel loss function and network architecture tailored for hallucination rather than standard correspondence matching.
• To demonstrate generalization to novel scenes not seen during training.

Proposed method

• The method, NeurHal, takes as input a pair of partially overlapping source and target images and a set of keypoints in the source image.
• It outputs a probability distribution over the target image plane for each keypoint’s correspondent, modeling both visible, occluded, and out-of-view cases.
• The network is trained with a novel loss function that encourages peaked distributions centered on the true correspondent location, regardless of visibility.
• The architecture is designed to handle both inpainting (occluded) and outpainting (outside field of view) scenarios through geometric reasoning.
• Training uses data from synthetic and real-world scenes, including ScanNet and MegaDepth, to enable zero-shot generalization.
• The approach leverages geometric priors implicitly through the training distribution and network inductive bias.

Experimental results

Research questions

• RQ1 Can a deep neural network be trained to hallucinate visual correspondences in occluded or out-of-view regions?
• RQ2 Is correspondence hallucination feasible and effective when generalizing to scenes not seen during training?
• RQ3 Does hallucinating correspondences improve robustness in absolute camera pose estimation compared to standard feature matching?
• RQ4 What architectural and loss function design choices are necessary to enable hallucination rather than standard matching?
• RQ5 How do failure cases correlate with visual ambiguity, low overlap, or strong camera rotations?

Key findings

• NeurHal successfully hallucinates correspondences on unseen scenes from ScanNet and MegaDepth, generalizing beyond training data.
• The network produces peaked probability distributions centered on true correspondents, even when they are occluded or outside the field of view.
• In absolute camera pose estimation, NeurHal outperforms state-of-the-art local feature matching methods, especially on low-overlap image pairs.
• Failure cases are primarily linked to extremely low visual overlap, strong camera rotations, and high image ambiguity.
• The method demonstrates robustness in both indoor (NYU) and outdoor (ETH3D) settings, with consistent performance across diverse scene types.
• Qualitative results show NeurHal effectively performs both inpainting (occluded) and outpainting (outside FOV) tasks, with accurate localization in challenging cases.