[Paper Review] Neural Matching Fields: Implicit Representation of Matching Fields for Visual Correspondence

Sunghwan Hong, Jisu Nam+5 more|arXiv (Cornell University)|Oct 6, 2022

Advanced Image and Video Retrieval Techniques4 citations

TL;DR

Neural Matching Fields (NeMF) proposes an implicit neural representation of 4D matching fields for high-precision semantic correspondence, using a cost embedding network with convolutions and self-attention to guide a fully-connected network in learning detailed correspondences. The method enables high-resolution inference via test-time optimization combining PatchMatch-based sampling and coordinate refinement, achieving state-of-the-art performance on standard benchmarks with improved localization accuracy over hand-crafted interpolation.

ABSTRACT

Existing pipelines of semantic correspondence commonly include extracting high-level semantic features for the invariance against intra-class variations and background clutters. This architecture, however, inevitably results in a low-resolution matching field that additionally requires an ad-hoc interpolation process as a post-processing for converting it into a high-resolution one, certainly limiting the overall performance of matching results. To overcome this, inspired by recent success of implicit neural representation, we present a novel method for semantic correspondence, called Neural Matching Field (NeMF). However, complicacy and high-dimensionality of a 4D matching field are the major hindrances, which we propose a cost embedding network to process a coarse cost volume to use as a guidance for establishing high-precision matching field through the following fully-connected network. Nevertheless, learning a high-dimensional matching field remains challenging mainly due to computational complexity, since a naive exhaustive inference would require querying from all pixels in the 4D space to infer pixel-wise correspondences. To overcome this, we propose adequate training and inference procedures, which in the training phase, we randomly sample matching candidates and in the inference phase, we iteratively performs PatchMatch-based inference and coordinate optimization at test time. With these combined, competitive results are attained on several standard benchmarks for semantic correspondence. Code and pre-trained weights are available at https://ku-cvlab.github.io/NeMF/.

Motivation & Objective

• To address the performance limitations of existing semantic correspondence methods that rely on low-resolution matching fields and hand-crafted interpolation.
• To overcome the high computational complexity and intractability of learning and inferring high-dimensional 4D matching fields.
• To enable high-resolution, precise correspondence prediction at original image resolution using implicit neural representation.
• To design a training and inference pipeline that efficiently handles the complexity and dimensionality of 4D matching fields.
• To validate the effectiveness of cost embedding and test-time optimization in improving correspondence accuracy.

Proposed method

• Proposes a cost embedding network using convolutions and self-attention layers to extract local and global context from a coarse cost volume, providing structured guidance for the implicit matching field.
• Employs a fully-connected neural network to implicitly represent the 4D matching field at arbitrary resolution, enabling continuous, high-precision correspondence prediction.
• Uses a sampling-based training strategy that randomly samples matching candidates and supervises the network on correspondence correctness, avoiding full inference during training.
• Introduces a test-time optimization procedure combining PatchMatch-based search space sampling and iterative coordinate optimization to refine predictions during inference.
• Alternates between exploration (PatchMatch-like sampling) and exploitation (coordinate optimization) to progressively improve correspondence accuracy at test time.
• Controls memory and inference time via resolution-aware design, enabling efficient inference even at high resolutions.

Experimental results

Research questions

• RQ1 Can implicit neural representation effectively model high-dimensional 4D matching fields for semantic correspondence?
• RQ2 How can cost volume features be effectively embedded to guide high-precision matching field learning?
• RQ3 Can sampling-based training and test-time optimization jointly overcome the computational intractability of 4D matching field inference?
• RQ4 Does the proposed method outperform hand-crafted interpolation and prior implicit or explicit matching methods on standard benchmarks?
• RQ5 What is the impact of cost embedding components and test-time optimization on final correspondence accuracy?

Key findings

• NeMF achieves state-of-the-art performance on standard semantic correspondence benchmarks, significantly outperforming methods using bilinear or TPS interpolation.
• The proposed test-time optimization strategy (PatchMatch + coordinate optimization) reduces inference time from over 300k seconds (exhaustive) to under 9 seconds on a single GPU, while improving accuracy.
• The method reduces memory consumption at high resolutions compared to prior methods like CHM and CATs, with 6,309 MB at 128×128 resolution, compared to OOM for others.
• Ablation studies confirm that both the cost embedding network and test-time optimization are essential for performance, with the latter showing a 20%+ improvement in accuracy over baseline inference.
• The training strategy enables stable learning without full-volume inference, making high-resolution training feasible despite computational constraints.
• The method maintains high performance across multiple datasets, demonstrating robustness to intra-class variations and geometric distortions.