[Paper Review] LoFTR: Detector-Free Local Feature Matching with Transformers

Jiaming Sun, Zehong Shen+3 more|arXiv (Cornell University)|Apr 1, 2021

Advanced Image and Video Retrieval Techniques42 references75 citations

TL;DR

LoFTR proposes detector-free, dense local feature matching using Transformer self- and cross-attention to establish dense correspondences, achieving state-of-the-art results on pose estimation and visual localization. It uses a coarse-to-fine refinement and a linear- Transformer variant for efficiency.

ABSTRACT

We present a novel method for local image feature matching. Instead of performing image feature detection, description, and matching sequentially, we propose to first establish pixel-wise dense matches at a coarse level and later refine the good matches at a fine level. In contrast to dense methods that use a cost volume to search correspondences, we use self and cross attention layers in Transformer to obtain feature descriptors that are conditioned on both images. The global receptive field provided by Transformer enables our method to produce dense matches in low-texture areas, where feature detectors usually struggle to produce repeatable interest points. The experiments on indoor and outdoor datasets show that LoFTR outperforms state-of-the-art methods by a large margin. LoFTR also ranks first on two public benchmarks of visual localization among the published methods.

Motivation & Objective

• Avoid the repeatability issues of traditional detector-based matching by eliminating the detector stage and performing dense matching directly.
• Leverage Transformer’s global receptive field to produce reliable matches in low-texture or repetitive regions.
• Introduce a coarse-to-fine matching pipeline with efficient attention for scalability.
• Demonstrate state-of-the-art performance on indoor and outdoor datasets for relative pose estimation and visual localization.

Proposed method

• Extract coarse and fine local features from CNN backbones with FPN from image pairs.
• Transform coarse features with a Local Feature Transformer (LoFTR) using interleaved self- and cross-attention layers.
• Use a linear attention variant to achieve O(N) complexity for attention.
• Compute a coarse-level match probability with either optimal transport or dual-softmax, then select high-confidence matches.
• Refine selected coarse matches to sub-pixel accuracy via a local window correlation-based refinement (coarse-to-fine).
• Supervise with coarse-level (confidence-based) and fine-level (L2 with uncertainty weighting) losses.

Figure 1: Comparison between the proposed method LoFTR and the detector-based method SuperGlue [ 37 ] . This example demonstrates that LoFTR is capable of finding correspondences on the texture-less wall and the floor with repetitive patterns, where detector-based methods struggle to find repeatable

Experimental results

Research questions

• RQ1 Can detector-free, dense matching via Transformers produce robust correspondences in low-texture or repetitive regions where detectors fail?
• RQ2 Does a coarse-to-fine, Transformer-based approach improve relative pose estimation and visual localization compared to detector-based and other detector-free methods?
• RQ3 What is the impact of using linear attention versus vanilla attention on performance and efficiency?
• RQ4 How do coarse-level and fine-level supervision contribute to overall matching accuracy?

Key findings

• LoFTR achieves state-of-the-art performance on multiple indoor and outdoor datasets for pose estimation and visual localization.
• The detector-free design yields high-quality matches in indistinctive regions and under challenging conditions (low texture, motion blur, repetitive patterns).
• Transformer-based global receptive fields enable dense matching beyond traditional detector-based priors, contributing to substantial performance gains over baselines.
• Coarse-to-fine refinement improves accuracy to sub-pixel localization.
• LoFTR with dual-softmax or optimal transport achieves strong results on benchmarks like HPatches, ScanNet, MegaDepth, and VisLoc; it ranks first on two public visual localization benchmarks.

Figure 2: Overview of the proposed method. LoFTR has four components: 1. A local feature CNN extracts the coarse-level feature maps $\tilde{F}^{A}$ and $\tilde{F}^{B}$ , together with the fine-level feature maps $\hat{F}^{A}$ and $\hat{F}^{B}$ from the image pair $I^{A}$ and $I^{B}$ (Section 3.1 ).