[Paper Review] LF-Net: Learning Local Features from Images

Y. Ono, Eduard Trulls+2 more|arXiv (Cornell University)|May 24, 2018

Image Retrieval and Classification Techniques223 citations

TL;DR

LF-Net presents a two-branch self-supervised architecture to learn a full local feature pipeline end-to-end from image pairs with known depth/pose, achieving state-of-the-art sparse matching and running at 60+ fps on QVGA images.

ABSTRACT

We present a novel deep architecture and a training strategy to learn a local feature pipeline from scratch, using collections of images without the need for human supervision. To do so we exploit depth and relative camera pose cues to create a virtual target that the network should achieve on one image, provided the outputs of the network for the other image. While this process is inherently non-differentiable, we show that we can optimize the network in a two-branch setup by confining it to one branch, while preserving differentiability in the other. We train our method on both indoor and outdoor datasets, with depth data from 3D sensors for the former, and depth estimates from an off-the-shelf Structure-from-Motion solution for the latter. Our models outperform the state of the art on sparse feature matching on both datasets, while running at 60+ fps for QVGA images.

Motivation & Objective

• Learn a local feature extraction pipeline from scratch without hand-crafted detectors.
• Use depth and relative pose to create virtual supervision targets for training.
• Combine differentiable and non-differentiable branches to enable end-to-end learning.
• Demonstrate strong sparse-matching performance on indoor and outdoor datasets.
• Show fast inference suitable for real-time applications.

Proposed method

• LF-Net comprises a detector that outputs a scale-space score map, dense orientations, and a descriptor network for patches around keypoints.
• Training uses two identical network branches processing two images; the right branch provides a non-differentiable supervision signal to the left branch via warping with ground-truth geometry.
• A differentiable sampler (STN) crops patches around detected keypoints for descriptor learning.
• Scale-space keypoint detection uses multi-scale feature maps with soft non-maximum suppression and softargmax for sub-pixel accuracy.
• Orientation is predicted from the shared feature map via a 5x5 conv producing sine/cosine components.
• Descriptors are 256-D, L2-normalized, learned via a patch-based descriptor loss (triplet loss with hard-negative mining).
• The training losses include an image-level loss on warped score maps, a patch-level loss aligning descriptors across views, and a geometry loss enforcing scale/orientation consistency.

Experimental results

Research questions

• RQ1 Can LF-Net learn discriminative keypoints and descriptors end-to-end without hand-crafted detectors?
• RQ2 How can a non-differentiable, warping-based supervision be incorporated into end-to-end learning?
• RQ3 Does the LF-Net training strategy generalize to indoor and outdoor datasets with depth inputs?
• RQ4 What is the performance of LF-Net on sparse feature matching compared with traditional and learned baselines, and what is its runtime?
• RQ5 How does joint training of detector and descriptor affect overall matching performance?

Key findings

• LF-Net achieves state-of-the-art sparse feature matching on indoor (ScanNet) and outdoor (photo-tourism) datasets.
• On outdoor data, LF-Net outperforms SuperPoint by 9% relative and LIFT by 45% relative in matching score (with/without rotation-scale augmentation).
• On indoor data, LF-Net matches SuperPoint for short-baseline sequences and surpasses SURF by a substantial margin when rotation/scale are used.
• LF-Net runs at about 60 fps for QVGA (320x240) frames and scales to real-time performance on typical GPUs (e.g., Titan X).
• Jointly training detector and descriptor in the two-branch setup yields consistent gains over training them separately (e.g., outdoors ablations show ~7% relative improvement).
• The method is robust across indoor/outdoor settings, though depth noise can affect indoors where depth devices (e.g., Kinect) produce incomplete maps.