[Paper Review] Recurrent Transformer Networks for Semantic Correspondence

Seungryong Kim, Stephen Lin+3 more|arXiv (Cornell University)|Oct 29, 2018

Advanced Image and Video Retrieval TechniquesComputer Science36 references49 citations

TL;DR

RTNs directly estimate locally-varying geometric transformations between image pairs and iteratively align features to obtain dense semantic correspondence, trained with a weakly-supervised classification loss.

ABSTRACT

We present recurrent transformer networks (RTNs) for obtaining dense correspondences between semantically similar images. Our networks accomplish this through an iterative process of estimating spatial transformations between the input images and using these transformations to generate aligned convolutional activations. By directly estimating the transformations between an image pair, rather than employing spatial transformer networks to independently normalize each individual image, we show that greater accuracy can be achieved. This process is conducted in a recursive manner to refine both the transformation estimates and the feature representations. In addition, a technique is presented for weakly-supervised training of RTNs that is based on a proposed classification loss. With RTNs, state-of-the-art performance is attained on several benchmarks for semantic correspondence.

Motivation & Objective

• Motivate dense correspondence across semantically similar images with large intra-class variation.
• Propose a method to directly estimate pairwise geometric deformations rather than normalizing each image independently.
• Develop a recurrent architecture that refines transformation estimates and feature representations across iterations.
• Enable weakly-supervised training without ground-truth transformation maps via a classification loss between source and geometrically aligned target features.

Proposed method

• Split RTNs into a feature extraction network and a geometric matching network with shared weights.
• Compute a correlation volume over a local search window to estimate dense affine transformation fields.
• Iteratively predict residual transformation fields to refine geometry and aligned features (K_max iterations).
• Transform target features using the current estimated geometry and extract updated features efficiently by reusing features from the whole image.
• Train with a weakly-supervised classification loss that encourages highest similarity for correct transformations and lower for others.

Experimental results

Research questions

• RQ1 Can RTNs learn locally-varying affine fields directly from image pairs without ground-truth transformations?
• RQ2 Does recursive refinement of geometry and features improve semantic correspondence accuracy over single-pass methods?
• RQ3 How does weak supervision via a classification loss influence learning of geometric fields and feature representations?

Key findings

• RTNs achieve state-of-the-art performance on semantic correspondence benchmarks compared to both weakly-supervised and supervised methods.
• The method converges within 3–5 iterations on tested datasets, with larger search windows improving accuracy up to a point.
• Jointly learned feature extraction and geometric matching networks outperform fixed pre-trained backbones.
• Locally-varying geometry estimation yields more precise localization than globally-varying approaches.