[Paper Review] RANSAC-Flow: generic two-stage image alignment

Xi Shen, François Darmon+2 more|arXiv (Cornell University)|Apr 3, 2020

Advanced Vision and Imaging97 references4 citations

TL;DR

RANSAC-Flow proposes a generic two-stage unsupervised method for dense image alignment by combining RANSAC-based homography estimation with a self-supervised deep network that refines alignment using SSIM and cycle-consistency loss. It achieves competitive performance across diverse tasks—including optical flow, visual localization, and artwork alignment—enabling new applications in art history and texture transfer.

ABSTRACT

This paper considers the generic problem of dense alignment between two images, whether they be two frames of a video, two widely different views of a scene, two paintings depicting similar content, etc. Whereas each such task is typically addressed with a domain-specific solution, we show that a simple unsupervised approach performs surprisingly well across a range of tasks. Our main insight is that parametric and non-parametric alignment methods have complementary strengths. We propose a two-stage process: first, a feature-based parametric coarse alignment using one or more homographies, followed by non-parametric fine pixel-wise alignment. Coarse alignment is performed using RANSAC on off-the-shelf deep features. Fine alignment is learned in an unsupervised way by a deep network which optimizes a standard structural similarity metric (SSIM) between the two images, plus cycle-consistency. Despite its simplicity, our method shows competitive results on a range of tasks and datasets, including unsupervised optical flow on KITTI, dense correspondences on Hpatches, two-view geometry estimation on YFCC100M, localization on Aachen Day-Night, and, for the first time, fine alignment of artworks on the Brughel dataset. Our code and data are available at http://imagine.enpc.fr/~shenx/RANSAC-Flow/

Motivation & Objective

• Address the challenge of generic dense image alignment across diverse image pairs with large appearance and viewpoint variations.
• Overcome limitations of purely parametric (e.g., RANSAC with homography) and non-parametric (e.g., optical flow) methods by combining their strengths.
• Enable unsupervised alignment without ground-truth flow or correspondences, particularly for rare or artistic image pairs.
• Support new applications in art history, such as analyzing copy processes in historical artworks, by enabling fine-grained alignment of visually dissimilar images.
• Provide a plug-and-play solution that works across multiple datasets and tasks without task-specific retraining.

Proposed method

• Stage 1: Estimate one or more coarse homographies using RANSAC on off-the-shelf deep features (e.g., from SuperPoint) to handle large viewpoint and appearance changes.
• Stage 2: Train a deep flow network in a self-supervised manner to predict dense pixel-wise flow fields that minimize SSIM between warped and target images.
• Enforce cycle-consistency loss to ensure that forward and backward warping produce consistent results, improving robustness.
• Use multiple homographies to model piecewise-planar scene structures, improving alignment accuracy over single-homography approaches.
• Iteratively refine alignment by removing matched correspondences and re-running the process to handle complex, non-uniform transformations.
• Leverage pre-trained features and a differentiable flow network to enable end-to-end optimization without ground-truth supervision.

Experimental results

Research questions

• RQ1 Can a simple, generic two-stage approach combining parametric and non-parametric methods achieve competitive performance across diverse image alignment tasks without task-specific design?
• RQ2 How effective is RANSAC with off-the-shelf deep features for coarse alignment in scenarios with large appearance and viewpoint differences?
• RQ3 To what extent can a self-supervised deep network trained on SSIM and cycle-consistency loss achieve high-precision dense alignment without ground-truth flow?
• RQ4 Can this method enable new applications in art history, such as analyzing the copy process in historical artworks with visually dissimilar versions?
• RQ5 Does using multiple homographies significantly improve alignment accuracy compared to a single homography in complex scenes?

Key findings

• RANSAC-Flow achieves competitive performance on unsupervised optical flow estimation on KITTI and Hpatches, outperforming or matching state-of-the-art methods without supervision.
• The method enables accurate visual localization on the Aachen Day-Night benchmark, with results on par with state-of-the-art methods using only image matching and COLMAP.
• It enables 2-view geometry estimation on YFCC100M using only image pairs, demonstrating robustness to large appearance and viewpoint variations.
• The approach achieves high-quality dense alignment on the Brughel dataset, enabling the first fine-grained analysis of artistic copy processes through flow visualization.
• The fine flow fields reveal spatially consistent copying in some artworks and discontinuous, irregular displacements in others, providing new insights into artistic techniques.
• The method enables effective texture transfer and internet image averaging, producing sharper and more coherent results than baseline alignment methods.