[Paper Review] AutoScaler: Scale-Attention Networks for Visual Correspondence

Shenlong Wang, Linjie Luo+2 more|arXiv (Cornell University)|Nov 17, 2016

Advanced Image and Video Retrieval Techniques40 references3 citations

TL;DR

AutoScaler proposes a scale-attention network that adaptively combines multi-scale feature maps in a learnable, end-to-end manner to improve visual correspondence accuracy. By using a shared-weight feature network and a scale-attention mechanism, it achieves state-of-the-art performance on optical flow and semantic matching benchmarks, including Sintel, KITTI, and CUB-2011, with interpretable attention maps highlighting optimal scale usage per region.

ABSTRACT

Finding visual correspondence between local features is key to many computer vision problems. While defining features with larger contextual scales usually implies greater discriminativeness, it could also lead to less spatial accuracy of the features. We propose AutoScaler, a scale-attention network to explicitly optimize this trade-off in visual correspondence tasks. Our network consists of a weight-sharing feature network to compute multi-scale feature maps and an attention network to combine them optimally in the scale space. This allows our network to have adaptive receptive field sizes over different scales of the input. The entire network is trained end-to-end in a siamese framework for visual correspondence tasks. Our method achieves favorable results compared to state-of-the-art methods on challenging optical flow and semantic matching benchmarks, including Sintel, KITTI and CUB-2011. We also show that our method can generalize to improve hand-crafted descriptors (e.g Daisy) on general visual correspondence tasks. Finally, our attention network can generate visually interpretable scale attention maps.

Motivation & Objective

• To address the trade-off between spatial accuracy and contextual discriminativeness in visual correspondence by optimizing scale selection.
• To enable adaptive receptive field sizes across different image regions through a learnable attention mechanism over multi-scale features.
• To improve performance on dense correspondence tasks such as optical flow and semantic matching without relying on handcrafted scale selection rules.
• To generalize to enhance hand-crafted descriptors like Daisy on general correspondence tasks.
• To generate visually interpretable scale attention maps that reveal where and why different scales are selected.

Proposed method

• The network uses a weight-shared feature extractor to compute multi-scale feature maps from the input image at multiple resolutions.
• An attention network predicts a pixel-wise attention map based on the input image, dynamically weighting the contribution of each scale.
• The final feature map is formed by a weighted sum of multi-scale features using the predicted attention map, enabling adaptive receptive fields.
• The entire model is trained end-to-end in a siamese framework using contrastive loss for correspondence learning.
• The attention mechanism is trained without explicit supervision on scale labels, learning optimal scale combinations from data.
• The method generalizes to improve hand-crafted descriptors by replacing their fixed-scale features with the learned multi-scale features.

Experimental results

Research questions

• RQ1 Can a deep learning model learn to optimally combine multi-scale features for visual correspondence without relying on heuristic scale selection?
• RQ2 How does adaptive scale weighting improve correspondence accuracy across diverse scenes and object scales?
• RQ3 Can the attention mechanism generate interpretable maps that reflect the network's scale selection strategy?
• RQ4 Does the proposed method generalize to improve existing hand-crafted descriptors like Daisy?
• RQ5 How does the model perform on challenging benchmarks such as Sintel, KITTI, and CUB-2011 compared to state-of-the-art methods?

Key findings

• On the Sintel benchmark, AutoScaler achieves state-of-the-art performance in both clean and final pass evaluations, outperforming prior methods in sub-pixel accuracy.
• On the KITTI 2015 optical flow benchmark, AutoScaler achieves 25.64% outlier rate on all pixels, ranking favorably among deep learning-based methods.
• On the CUB-2011 semantic matching dataset, AutoScaler achieves the highest PCK@α for small thresholds (α < 0.5), indicating superior sub-pixel accuracy.
• For larger thresholds (α ≥ 1.0), AutoScaler ranks second, behind only the Universal Correspondence Network, showing strong generalization in semantic reasoning.
• The attention maps generated by the network are visually interpretable, showing higher attention on fine scales in textured regions and coarser scales in textureless areas.
• The method improves the performance of hand-crafted descriptors such as Daisy when used with the learned multi-scale features, demonstrating strong generalization capability.