[Paper Review] PN-Net: Conjoined Triple Deep Network for Learning Local Image Descriptors

Vassileios Balntas, Edward Johns+2 more|arXiv (Cornell University)|Jan 19, 2016

Advanced Image and Video Retrieval TechniquesComputer Science17 references155 citations

TL;DR

PN-Net learns compact local image descriptors via a conjoined triplet CNN with the SoftPN loss, achieving fast extraction and improved matching accuracy over prior CNN-based descriptors. It trains quickly (≈2 minutes per epoch) and uses low-dimensional (≈128-D) descriptors suitable for fast nearest-neighbor matching.

ABSTRACT

In this paper we propose a new approach for learning local descriptors for matching image patches. It has recently been demonstrated that descriptors based on convolutional neural networks (CNN) can significantly improve the matching performance. Unfortunately their computational complexity is prohibitive for any practical application. We address this problem and propose a CNN based descriptor with improved matching performance, significantly reduced training and execution time, as well as low dimensionality. We propose to train the network with triplets of patches that include a positive and negative pairs. To that end we introduce a new loss function that exploits the relations within the triplets. We compare our approach to recently introduced MatchNet and DeepCompare and demonstrate the advantages of our descriptor in terms of performance, memory footprint and speed i.e. when run in GPU, the extraction time of our 128 dimensional feature is comparable to the fastest available binary descriptors such as BRIEF and ORB.

Motivation & Objective

• Motivate learning CNN-based local descriptors with reduced training and execution time.
• Develop a triplet-based network (PN-Net) that leverages positive/negative constraints within triplets.
• Introduce the SoftPN loss to exploit both positive and negative relations without heavy hard-negative mining.
• Achieve competitive or superior matching performance with low-dimensional descriptors and fast inference.

Proposed method

• Use a two-branch CNN that computes a D(p) descriptor for image patches without an explicit distance metric layer.
• Train on triplets {p1, p2, n} where p1 and p2 are from the same 3D point and n from a different point.
• Introduce SoftPN loss that encourages the smallest negative distance in the triplet to exceed the positive distance.
• Compare against siamese-based approaches (e.g., MatchNet, DeepCompare) using the same underlying CNN.
• Provide a compact network (two convolutional layers) that yields 128- or 256-D descriptors and is GPU-accelerated.
• No data augmentation used; training uses on-the-fly triplets from Patch data (Liberty/Yosemite/Notredame).

Experimental results

Research questions

• RQ1 Does a triplet-based CNN with SoftPN loss improve local descriptor matching over pair-based (siamese) CNN approaches?
• RQ2 Can PN-Net achieve competitive matching accuracy with low-dimensional descriptors and fast training/extraction times?
• RQ3 How does the proposed SoftPN loss compare to hinge embedding and SoftMax ratio in terms of convergence and performance?
• RQ4 What are the generalization properties of PN-Net across datasets (Liberty, Notredame, Yosemite) and the Oxford benchmark?

Key findings

• SoftPN loss with triplet training improves matching accuracy over hinge loss and SoftMax-based triplet losses.
• PN-Net achieves state-of-the-art or competitive performance while producing compact 128- or 256-D descriptors.
• Training is fast (≈2 minutes per epoch on a Titan X) and descriptor extraction on GPU is comparable to fast binary descriptors in speed.
• The approach reduces memory footprint and training time relative to other CNN-based descriptors while maintaining or improving accuracy.
• Generalization across datasets (Liberty, Notredame, Oxford) is strong, with good cross-dataset transfer observed.
• The method enables real-time capable descriptor extraction suitable for large-scale matching tasks.