[Paper Review] LIFT: Learned Invariant Feature Transform

Kwang Moo Yi, Eduard Trulls+2 more|arXiv (Cornell University)|Mar 30, 2016

Advanced Image and Video Retrieval TechniquesComputer Science37 references102 citations

TL;DR

LIFT proposes a fully differentiable, end-to-end deep network that jointly learns feature point detection, orientation estimation, and descriptor extraction, and it outperforms state-of-the-art methods on standard benchmarks.

ABSTRACT

We introduce a novel Deep Network architecture that implements the full feature point handling pipeline, that is, detection, orientation estimation, and feature description. While previous works have successfully tackled each one of these problems individually, we show how to learn to do all three in a unified manner while preserving end-to-end differentiability. We then demonstrate that our Deep pipeline outperforms state-of-the-art methods on a number of benchmark datasets, without the need of retraining.

Motivation & Objective

• Motivate the need for jointly learning detection, orientation, and description of local features within a single differentiable pipeline.
• Develop a three-component CNN-based architecture (Detector, Orientation Estimator, Descriptor) connected with differentiable operations to enable end-to-end training.
• Show that joint optimization yields better overall matching performance than optimizing components separately.
• Evaluate generalization of the learned features across diverse datasets with varying viewpoints and illumination.

Proposed method

• Introduce LIFT architecture with three CNN-based components: Detector, Orientation Estimator, and Descriptor.
• Use Spatial Transformer Layers to rectify image patches and preserve differentiability during cropping and rotation.
• Replace non-local maximum suppression with a differentiable soft argmax for end-to-end training.
• Train via a problem-specific schedule: first learn Descriptor, then Orientation Estimator, then Detector, using a Siamese setup with SfM-derived ground-truth correspondences.
• Employ a four-branch Siamese network on image patches corresponding to same/different 3D points to optimize descriptor, orientation, and detector jointly in the later stages.
• Evaluate against a broad set of baselines on Strecha, DTU, and Webcam datasets using repeatability, NN mAP, and matching score.

Experimental results

Research questions

• RQ1 Can a single differentiable network learn detection, orientation, and description that jointly optimize for robust cross-view matching?
• RQ2 Does end-to-end training of all three components yield better overall matching performance than tuning components in isolation?
• RQ3 How well do learned features generalize across datasets with different scenes, viewpoints, and illumination?

Key findings

• The integrated LIFT pipeline outperforms state-of-the-art baselines on multiple standard datasets.
• Each component (Detector, Orientation Estimator, Descriptor) contributes to overall performance, and jointly learned components surpass hand-crafted or separately learned counterparts.
• Soft argmax-based NMS and Spatial Transformer-based patch rectification preserve differentiability and enable end-to-end training.
• The Descriptor trained first, followed by Orientation Estimator and Detector, yields a practical and effective training strategy.
• The learned detectors generalize beyond the training datasets, performing robustly on scenes from DTU and the Webcam dataset despite dataset shifts.
• Ablation studies indicate replacing components with traditional SIFT-based or other baselines reduces performance, underscoring the benefit of the end-to-end learned pipeline.