[Paper Review] Learning to Compose Hypercolumns for Visual Correspondence

Juhong Min, Jong‐Min Lee+2 more|arXiv (Cornell University)|Jul 21, 2020

Advanced Image and Video Retrieval Techniques55 references4 citations

TL;DR

This paper proposes Dynamic Hyperpixel Flow, a novel method for visual correspondence that dynamically composes hypercolumn features by learning to select a small set of relevant convolutional layers from a deep CNN based on the input image pair. By leveraging Gumbel-softmax sampling for differentiable layer selection and adaptive feature fusion, the model achieves state-of-the-art performance on semantic correspondence benchmarks with improved accuracy and efficiency over static, monolithic feature representations.

ABSTRACT

Feature representation plays a crucial role in visual correspondence, and recent methods for image matching resort to deeply stacked convolutional layers. These models, however, are both monolithic and static in the sense that they typically use a specific level of features, e.g., the output of the last layer, and adhere to it regardless of the images to match. In this work, we introduce a novel approach to visual correspondence that dynamically composes effective features by leveraging relevant layers conditioned on the images to match. Inspired by both multi-layer feature composition in object detection and adaptive inference architectures in classification, the proposed method, dubbed Dynamic Hyperpixel Flow, learns to compose hypercolumn features on the fly by selecting a small number of relevant layers from a deep convolutional neural network. We demonstrate the effectiveness on the task of semantic correspondence, i.e., establishing correspondences between images depicting different instances of the same object or scene category. Experiments on standard benchmarks show that the proposed method greatly improves matching performance over the state of the art in an adaptive and efficient manner.

Motivation & Objective

• To address the limitations of monolithic and static deep feature representations in visual correspondence tasks.
• To enable adaptive feature composition by selecting relevant CNN layers based on input image pairs.
• To improve matching accuracy under large intra-class variations, such as viewpoint changes and deformations.
• To develop an efficient, trainable architecture that dynamically fuses multi-level features without relying on fixed layer selection.
• To demonstrate effectiveness in both strongly-supervised and weakly-supervised learning regimes.

Proposed method

• The method uses a Gumbel-softmax differentiable sampling strategy to learn discrete layer selection across multiple convolutional layers in a CNN.
• It formulates dynamic feature composition as a gating mechanism that selects a small subset of layers per image pair, enabling adaptive fusion of multi-level features.
• The model employs a learnable gating network that predicts layer weights using a multi-layer perceptron (MLP) conditioned on the input image pair.
• Hypercolumn features are constructed by concatenating activations from selected layers, with dimensionality reduced via a learnable feature transformation (CFT) layer.
• A novel loss function, $\mathcal{L}_{\text{sel}}$, encourages diversity in layer selection during training to avoid overfitting to specific layers.
• The architecture is trained end-to-end with a combination of correspondence loss and layer selection regularization, enabling efficient inference.

Experimental results

Research questions

• RQ1 Can dynamic, learnable layer selection improve visual correspondence performance compared to fixed, monolithic feature representations?
• RQ2 How does adaptive composition of multi-level features from a deep CNN affect accuracy and efficiency in semantic correspondence?
• RQ3 What is the impact of differentiable, stochastic layer selection (via Gumbel-softmax) versus soft gating on model performance and generalization?
• RQ4 Can the proposed method generalize effectively in both strongly-supervised and weakly-supervised settings?
• RQ5 How does the computational cost of dynamic feature composition compare to existing methods with similar or better accuracy?

Key findings

• The proposed Dynamic Hyperpixel Flow achieves state-of-the-art performance on the PF-PASCAL benchmark, with a PCK score of 75.7 at α=0.05, surpassing previous methods.
• On the PASCAL-Part dataset, the method achieves 73.6 PCK at α=0.05, demonstrating strong generalization to fine-grained part-level correspondence.
• The Gumbel-softmax gating mechanism outperforms all soft-gating variants (e.g., sigmoid, sigmoid with ℓ1 regularization), achieving higher accuracy and faster inference (58ms vs. 74ms).
• The model reduces effective feature dimensionality to 3962 on average (for μ=0.5), significantly lower than prior methods (e.g., 6400–10240), improving efficiency.
• Ablation studies show that removing the layer selection loss ($\mathcal{L}_{\text{sel}}$) or the margin loss ($\omega_m$) leads to significant performance drops, confirming the importance of the proposed training objectives.
• Layer selection frequency analysis confirms that the model learns diverse, non-uniform layer selection patterns, indicating effective exploration of multi-level representations.