[Paper Review] Integrative Feature and Cost Aggregation with Transformers for Dense Correspondence

Sunghwan Hong, Seokju Cho+2 more|arXiv (Cornell University)|Sep 19, 2022

Advanced Image and Video Retrieval Techniques4 citations

TL;DR

This paper proposes IFCAT, a novel Transformer-based architecture that jointly aggregates feature descriptors and cost volumes in a complementary, interleaved manner to improve dense correspondence. By using self-attention to refine features with noisy cost volume guidance and cross-attention to enhance matching via bidirectional feature and cost supervision, IFCAT achieves state-of-the-art performance on SPair-71k, PF-PASCAL, PF-WILLOW, and HPatches, with 64.4% on SPair-71k and 17.59 AEPE on HPatches.

ABSTRACT

We present a novel architecture for dense correspondence. The current state-of-the-art are Transformer-based approaches that focus on either feature descriptors or cost volume aggregation. However, they generally aggregate one or the other but not both, though joint aggregation would boost each other by providing information that one has but other lacks, i.e., structural or semantic information of an image, or pixel-wise matching similarity. In this work, we propose a novel Transformer-based network that interleaves both forms of aggregations in a way that exploits their complementary information. Specifically, we design a self-attention layer that leverages the descriptor to disambiguate the noisy cost volume and that also utilizes the cost volume to aggregate features in a manner that promotes accurate matching. A subsequent cross-attention layer performs further aggregation conditioned on the descriptors of both images and aided by the aggregated outputs of earlier layers. We further boost the performance with hierarchical processing, in which coarser level aggregations guide those at finer levels. We evaluate the effectiveness of the proposed method on dense matching tasks and achieve state-of-the-art performance on all the major benchmarks. Extensive ablation studies are also provided to validate our design choices.

Motivation & Objective

• Address the limitation of existing Transformer-based dense correspondence methods that perform either feature aggregation or cost aggregation, but not both in a synergistic way.
• Leverage the complementary strengths of feature descriptors (semantic/structural context) and cost volumes (pixel-wise matching similarity) to improve matching accuracy.
• Design a joint aggregation mechanism that enables mutual enhancement between feature and cost volume through interleaved self- and cross-attention layers.
• Improve performance through hierarchical processing, where coarser-level outputs guide finer-level aggregation.
• Establish a new state-of-the-art on multiple standard benchmarks for both geometric and semantic matching tasks.

Proposed method

• Propose a self-attention layer that uses feature descriptors to disambiguate noisy cost volumes and uses the cost volume to guide feature aggregation by incorporating matching similarity.
• Introduce a cross-attention layer that conditions on descriptors from both images and is aided by aggregated features and cost volume from earlier layers to further refine representations.
• Interleave self- and cross-attention blocks to enable iterative refinement and mutual enhancement of features and cost volume.
• Implement hierarchical processing where coarser-level feature and cost volume outputs guide subsequent finer-level aggregations.
• Design the architecture to be computationally efficient while maintaining high performance, with ablation studies on depth and efficiency metrics.
• Train the model end-to-end with a learnable loss function optimized for AEPE and bbox accuracy on matching benchmarks.

Experimental results

Research questions

• RQ1 Can joint aggregation of feature descriptors and cost volumes in a Transformer-based framework improve dense correspondence performance beyond sequential or isolated processing?
• RQ2 How does the integration of feature and cost volume information through self- and cross-attention enhance matching accuracy and robustness?
• RQ3 To what extent does hierarchical processing improve the performance of integrative aggregation in dense correspondence?
• RQ4 What is the contribution of each architectural component—self-attention, cross-attention, and hierarchical design—to the final performance gain?
• RQ5 How does the proposed method compare to existing SOTA approaches in terms of accuracy and efficiency across diverse benchmarks?

Key findings

• IFCAT achieves 64.4% on SPair-71k (α_bbox = 0.1) and 17.59 AEPE on HPatches, establishing a new state-of-the-art on all evaluated benchmarks.
• The ablation study shows that solely aggregating features or cost volumes yields limited performance (e.g., 36.1% on SPair-71k for feature self-attention only), but joint aggregation significantly improves results.
• The integrative self-attention layer (54.7% on SPair-71k) and subsequent cross-attention (58.5%) progressively boost performance, demonstrating the value of complementary information flow.
• Hierarchical processing further improves performance from 58.5% to 64.4% on SPair-71k and from 24.41 to 17.59 AEPE on HPatches, confirming the benefit of coarse-to-fine guidance.
• Increasing the depth of the attention block (N) improves performance monotonically (e.g., 55.4% to 64.7% on SPair-71k), though with increasing memory and runtime costs.
• The method outperforms two-stage RANSAC-based methods on HPatches when used independently, showing strong generalization without relying on homography estimation.