[Paper Review] Parallax Attention for Unsupervised Stereo Correspondence Learning

Longguang Wang, Yulan Guo+5 more|arXiv (Cornell University)|Sep 16, 2020

Advanced Vision and Imaging70 references7 citations

TL;DR

This paper proposes a generic parallax-attention mechanism (PAM) that learns stereo correspondence under large disparity variations without requiring a fixed maximum disparity, using epipolar constraints and attention-based feature matching. PAM enables state-of-the-art performance in unsupervised stereo matching (PASMnet) and stereo image super-resolution (PASSRnet), with a new large-scale dataset, Flickr1024, supporting training and evaluation.

ABSTRACT

Stereo image pairs encode 3D scene cues into stereo correspondences between the left and right images. To exploit 3D cues within stereo images, recent CNN based methods commonly use cost volume techniques to capture stereo correspondence over large disparities. However, since disparities can vary significantly for stereo cameras with different baselines, focal lengths and resolutions, the fixed maximum disparity used in cost volume techniques hinders them to handle different stereo image pairs with large disparity variations. In this paper, we propose a generic parallax-attention mechanism (PAM) to capture stereo correspondence regardless of disparity variations. Our PAM integrates epipolar constraints with attention mechanism to calculate feature similarities along the epipolar line to capture stereo correspondence. Based on our PAM, we propose a parallax-attention stereo matching network (PASMnet) and a parallax-attention stereo image super-resolution network (PASSRnet) for stereo matching and stereo image super-resolution tasks. Moreover, we introduce a new and large-scale dataset named Flickr1024 for stereo image super-resolution. Experimental results show that our PAM is generic and can effectively learn stereo correspondence under large disparity variations in an unsupervised manner. Comparative results show that our PASMnet and PASSRnet achieve the state-of-the-art performance.

Motivation & Objective

• To address the challenge of handling large disparity variations in stereo matching due to varying camera baselines, focal lengths, and resolutions.
• To overcome the limitations of fixed maximum disparity in cost volume-based methods that hinder performance on diverse stereo image pairs.
• To develop a generic, compact attention mechanism that integrates epipolar geometry with feature attention for accurate stereo correspondence.
• To extend the proposed method to multiple tasks, including stereo matching and stereo image super-resolution, achieving state-of-the-art results.

Proposed method

• Proposes a parallax-attention mechanism (PAM) that computes feature similarities along epipolar lines using matrix multiplication instead of shift operations.
• Integrates epipolar constraints into an attention mechanism to focus on the most similar features across left and right images, enabling robust matching under large disparities.
• Uses direct regularization on attention maps via smoothness and cycle consistency losses, avoiding ambiguity in disparity regression.
• Applies PAM to build PASMnet for unsupervised stereo matching and PASSRnet for stereo image super-resolution, both trained without ground-truth depth.
• Employs a novel large-scale dataset, Flickr1024, with 1024 high-quality stereo pairs covering diverse scenes and disparities for training and evaluation.
• Leverages matrix operations to avoid explicit disparity regression, enabling end-to-end learning without hard-coded disparity range limits.

Experimental results

Research questions

• RQ1 Can a deep learning method learn stereo correspondence robustly across large disparity variations without relying on a fixed maximum disparity?
• RQ2 How does integrating epipolar geometry with attention mechanisms improve stereo matching accuracy in an unsupervised setting?
• RQ3 Can the proposed parallax-attention mechanism be generalized across different stereo vision tasks such as stereo matching and super-resolution?
• RQ4 To what extent does the absence of a fixed disparity range improve performance on stereo image pairs with extreme disparities?
• RQ5 How does the proposed method compare to existing cost volume-based approaches in terms of efficiency, accuracy, and generalization across diverse camera configurations?

Key findings

• PASSRnet achieves a PSNR of 39.03 dB on stereo image pairs with small depths, outperforming StereoSR by 1.43 dB.
• On large-baseline stereo pairs, PASSRnet improves PSNR by 1.30 dB over StereoSR, demonstrating robustness to large disparities.
• PASMnet achieves state-of-the-art performance in unsupervised stereo matching, with a 1.20 dB improvement in PSNR over baseline methods on the KITTI dataset.
• PASSRnet reduces computational cost and avoids unnecessary padding on low-resolution images, showing higher efficiency than StereoSR.
• The proposed Flickr1024 dataset enables effective training and evaluation, supporting diverse scene types and disparity ranges.
• The ablation study confirms that direct regularization on attention maps improves performance over traditional disparity regression, reducing ambiguity in cost distribution.