[Paper Review] Convolutional neural network architecture for geometric matching

Ignacio Rocco, Relja Arandjelović+1 more|HAL (Le Centre pour la Communication Scientifique Directe)|Mar 16, 2017

Advanced Image and Video Retrieval TechniquesComputer Science47 references53 citations

TL;DR

A fully trainable CNN architecture for estimating geometric transformations between image pairs, using a differentiable matching layer and end-to-end training on synthetic data to achieve state-of-the-art results on instance- and category-level matching.

ABSTRACT

We address the problem of determining correspondences between two images in agreement with a geometric model such as an affine or thin-plate spline transformation, and estimating its parameters. The contributions of this work are three-fold. First, we propose a convolutional neural network architecture for geometric matching. The architecture is based on three main components that mimic the standard steps of feature extraction, matching and simultaneous inlier detection and model parameter estimation, while being trainable end-to-end. Second, we demonstrate that the network parameters can be trained from synthetically generated imagery without the need for manual annotation and that our matching layer significantly increases generalization capabilities to never seen before images. Finally, we show that the same model can perform both instance-level and category-level matching giving state-of-the-art results on the challenging Proposal Flow dataset.

Motivation & Objective

• Motivate robust image-to-image correspondence under large appearance changes and clutter.
• Develop an end-to-end trainable architecture that mimics classical feature matching and robust transformation estimation.
• Enable training from synthetic data without manual annotations while achieving strong generalization.
• Support both instance-level and category-level geometric matching within a single framework.

Proposed method

• Siamese CNN feature extractor (VGG-16 up to pool4, with per-feature L2 normalization) to produce dense descriptors.
• Correlation-based matching layer computes all pairwise descriptor similarities and applies channel-wise normalization to down-weight ambiguous matches.
• Regression network (two convolutional layers + batch norm + ReLU + final fully connected layer) estimates transformation parameters from the normalized correlation map.
• Hierarchical transformation modeling: first estimate an affine transformation, warp image A, then estimate a thin-plate spline (TPS) for refinement, composing to the final TPS.
• Training from synthetic transformations using fully supervised losses on a deformed grid, enabling end-to-end backpropagation without manual annotations.
• Loss function measures grid-point displacement after applying ground-truth and estimated transformations on a deformable grid; gradients are computed with respect to transformation parameters.

Experimental results

Research questions

• RQ1 Can a fully trainable CNN architecture mimic classical matching pipelines to estimate geometric transformations between image pairs?
• RQ2 Does a correlation-based matching layer with normalization improve robustness to clutter and repetitive patterns compared to concatenation or subtraction schemes?
• RQ3 Can end-to-end training from synthetic data generalize to unseen images and support both category-level and instance-level matching?
• RQ4 What gains arise from a progressive, two-stage estimation (affine then TPS) for complex geometric transformations?

Key findings

• The proposed method achieves state-of-the-art PCK on the Proposal Flow dataset, outperforming methods that rely on object proposals.
• Affine and TPS stages improve alignment, with ensemble affine estimates further boosting performance.
• The correlation-layer with channel-wise normalization outperforms concatenation or subtraction matching strategies in generalization and accuracy.
• Normalization (mimicking second nearest neighbor) significantly enhances performance (e.g., 44% to 49% in ablation).
• The approach generalizes well across training data sources (StreetView vs Pascal VOC synthetic datasets).
• Qualitative results demonstrate robust handling of large appearance changes, clutter, background, and non-rigid deformations.