[Paper Review] LATCH: Learned Arrangements of Three Patch Codes

Gil Levi, Tal Hassner|arXiv (Cornell University)|Jan 15, 2015

Advanced Image and Video Retrieval Techniques24 references4 citations

TL;DR

LATCH proposes a novel binary local image descriptor that improves robustness and performance by learning optimal arrangements of three-patch comparisons instead of traditional pixel-pair comparisons. By using supervised learning to select discriminative, non-correlated patch triplets, LATCH achieves state-of-the-art performance among binary descriptors while maintaining low computational and memory costs, significantly narrowing the gap with histogram-based methods like SIFT.

ABSTRACT

We present a novel means of describing local image appearances using binary strings. Binary descriptors have drawn increasing interest in recent years due to their speed and low memory footprint. A known shortcoming of these representations is their inferior performance compared to larger, histogram based descriptors such as the SIFT. Our goal is to close this performance gap while maintaining the benefits attributed to binary representations. To this end we propose the Learned Arrangements of Three Patch Codes descriptors, or LATCH. Our key observation is that existing binary descriptors are at an increased risk from noise and local appearance variations. This, as they compare the values of pixel pairs; changes to either of the pixels can easily lead to changes in descriptor values, hence damaging its performance. In order to provide more robustness, we instead propose a novel means of comparing pixel patches. This ostensibly small change, requires a substantial redesign of the descriptors themselves and how they are produced. Our resulting LATCH representation is rigorously compared to state-of-the-art binary descriptors and shown to provide far better performance for similar computation and space requirements.

Motivation & Objective

• To address the sensitivity of existing binary descriptors to noise and local appearance variations caused by pixel-pair comparisons.
• To improve the discriminative power of binary descriptors without increasing computational or storage costs.
• To develop a method that learns optimal, non-reduundant patch triplet arrangements for robust descriptor generation.
• To close the performance gap between binary descriptors and larger histogram-based descriptors like SIFT.
• To enable efficient, fast matching while maintaining high accuracy in real-world applications such as 3D reconstruction.

Proposed method

• LATCH replaces traditional pixel-pair comparisons with comparisons of three small image patches to increase spatial support and noise resilience.
• Patch triplets are selected via supervised learning on a labeled dataset of 'same' and 'not-same' patch pairs to maximize discriminative power.
• The learning process filters out highly correlated triplets to ensure diversity and improve descriptor robustness.
• Each bit in the final binary descriptor is generated by comparing intensity patterns across the three patches in a triplet.
• The method is trained off-line and applied during descriptor extraction using a pre-learned set of optimal triplet arrangements.
• The implementation is integrated into OpenCV for broad accessibility and reproducibility.

Experimental results

Research questions

• RQ1 Can replacing pixel-pair comparisons with three-patch comparisons significantly improve the robustness of binary descriptors to noise and local appearance changes?
• RQ2 Can supervised learning of patch triplet arrangements yield more discriminative and non-redundant descriptors than heuristic or random selection?
• RQ3 To what extent can a learned three-patch descriptor match or exceed the performance of SIFT while retaining binary descriptor efficiency?
• RQ4 How does LATCH perform in real-world applications such as structure from motion, where matching speed and accuracy are critical?
• RQ5 Can the performance gap between binary and histogram-based descriptors be meaningfully reduced through structural rethinking of the descriptor design?

Key findings

• LATCH outperforms all existing binary descriptors on the Oxford and Learning Local Descriptors benchmarks, with significant improvements in matching accuracy.
• In structure from motion tasks, LATCH achieves qualitatively comparable 3D reconstructions to SIFT while being an order of magnitude faster to match.
• Descriptor matching time for LATCH was 39.05 seconds on Sceaux Castle, compared to 381.63 seconds for SIFT, demonstrating a 10x speedup.
• The combined learning method—using both triplet and pair learning—outperformed both ORB-style pair learning and standalone triplet learning.
• Randomly selected triplets performed significantly worse than learned arrangements, confirming the value of supervised selection.
• LATCH maintains high performance even when the ratio test threshold is increased to 0.99, indicating greater robustness in binary descriptor matching.