[Paper Review] LIFE: Lighting Invariant Flow Estimation

Zhaoyang Huang, Xiaokun Pan+5 more|arXiv (Cornell University)|Apr 7, 2021

Advanced Vision and Imaging59 references5 citations

TL;DR

LIFE proposes a weakly supervised deep learning framework for estimating dense optical flow between image pairs with large lighting variations, using symmetric epipolar distance loss from Structure-from-Motion-derived camera poses and synthetic geometric transformations to guide flow prediction. It achieves state-of-the-art performance in challenging lighting conditions and significantly improves sparse feature matching and downstream visual localization tasks.

ABSTRACT

We tackle the problem of estimating flow between two images with large lighting variations. Recent learning-based flow estimation frameworks have shown remarkable performance on image pairs with small displacement and constant illuminations, but cannot work well on cases with large viewpoint change and lighting variations because of the lack of pixel-wise flow annotations for such cases. We observe that via the Structure-from-Motion (SfM) techniques, one can easily estimate relative camera poses between image pairs with large viewpoint change and lighting variations. We propose a novel weakly supervised framework LIFE to train a neural network for estimating accurate lighting-invariant flows between image pairs. Sparse correspondences are conventionally established via feature matching with descriptors encoding local image contents. However, local image contents are inevitably ambiguous and error-prone during the cross-image feature matching process, which hinders downstream tasks. We propose to guide feature matching with the flows predicted by LIFE, which addresses the ambiguous matching by utilizing abundant context information in the image pairs. We show that LIFE outperforms previous flow learning frameworks by large margins in challenging scenarios, consistently improves feature matching, and benefits downstream tasks.

Motivation & Objective

• To address the lack of dense flow annotations for image pairs with large lighting and viewpoint variations.
• To develop a weakly supervised learning framework that leverages camera pose estimates from Structure-from-Motion (SfM) as supervision for optical flow estimation.
• To improve sparse feature matching by guiding it with lighting-invariant flows predicted by the proposed network.
• To enhance downstream tasks such as visual localization and 3D reconstruction in real-world scenarios with extreme illumination changes.

Proposed method

• Leverages relative camera poses from SfM to compute symmetric epipolar distance as a weak supervision signal for flow estimation.
• Introduces a symmetric epipolar distance loss that constrains predicted flows to lie on epipolar lines, allowing for sliding along the line while preserving geometric consistency.
• Uses synthetic geometric transformations (e.g., homographies) to generate pixel-wise flow supervision, complementing the epipolar loss and improving accuracy.
• Applies bidirectional flow training (BiT loss) to enforce consistency in both directions, improving robustness to occlusions and ambiguous regions.
• Implements adaptive cycle consistency (AC) loss that applies cycle consistency only to low-error flow predictions, reducing error propagation in occluded areas.
• Guides sparse feature matching in two stages: first using standard descriptors, then refining matches using predicted flows from LIFE to reduce ambiguity and outliers.

Experimental results

Research questions

• RQ1 Can symmetric epipolar distance loss from SfM camera poses serve as an effective weak supervision signal for learning lighting-invariant optical flow?
• RQ2 How does combining synthetic geometric transformations with epipolar constraints improve flow estimation accuracy under large illumination variations?
• RQ3 To what extent can lighting-invariant flows improve the quality of sparse feature matching in terms of inlier ratio and match count?
• RQ4 Does the proposed framework improve downstream tasks such as visual localization in challenging lighting conditions?

Key findings

• LIFE achieves state-of-the-art performance on KITTI and HPatches benchmarks, reducing average end-point error (AEPE) to 2.59 on HPatches-I(T) and 8.30 on KITTI 2015.
• The combination of symmetric epipolar distance loss, BiT loss, and adaptive cycle consistency (SED+BiT+AC) yields the best performance, outperforming individual components.
• LIFE+R2D2 achieves 94.9% localization accuracy at 0.5m, 5° threshold on the Aachen nighttime benchmark, ranking first among all methods tested.
• LIFE improves sparse matching by increasing inlier ratio and match count: for example, R2D2 matches rise from 1.8K to 2.1K when using LIFE-guided matching.
• The use of LIFE-guided matching increases the match number while slightly reducing MMA, indicating that the supplementary matches are less accurate but more numerous, improving overall robustness.
• On the Aachen nighttime dataset, LIFE+R2D2 achieves 100% accuracy at 5m, 10° threshold, outperforming SuperGlue and other SOTA methods.