[Paper Review] FADNet++: Real-Time and Accurate Disparity Estimation with Configurable Networks

Qiang Wang, Shaohuai Shi+3 more|arXiv (Cornell University)|Oct 6, 2021

Advanced Vision and Imaging48 references4 citations

TL;DR

FADNet++ is a configurable, real-time deep neural network for disparity estimation that combines efficient 2D convolutions with point-wise correlation layers and residual learning to achieve state-of-the-art accuracy while maintaining sub-100ms inference speed across diverse hardware. It enables users to generate model variants (FADNet-T to FADNet++) with adjustable accuracy and speed for deployment on mobile and server GPUs.

ABSTRACT

Deep neural networks (DNNs) have achieved great success in the area of computer vision. The disparity estimation problem tends to be addressed by DNNs which achieve much better prediction accuracy than traditional hand-crafted feature-based methods. However, the existing DNNs hardly serve both efficient computation and rich expression capability, which makes them difficult for deployment in real-time and high-quality applications, especially on mobile devices. To this end, we propose an efficient, accurate, and configurable deep network for disparity estimation named FADNet++. Leveraging several liberal network design and training techniques, FADNet++ can boost its accuracy with a fast model inference speed for real-time applications. Besides, it enables users to easily configure different sizes of models for balancing accuracy and inference efficiency. We conduct extensive experiments to demonstrate the effectiveness of FADNet++ on both synthetic and realistic datasets among six GPU devices varying from server to mobile platforms. Experimental results show that FADNet++ and its variants achieve state-of-the-art prediction accuracy, and run at a significant order of magnitude faster speed than existing 3D models. With the constraint of running at above 15 frames per second (FPS) on a mobile GPU, FADNet++ achieves a new state-of-the-art result for the SceneFlow dataset.

Motivation & Objective

• Address the trade-off between inference speed and accuracy in deep learning-based stereo matching for real-time applications.
• Overcome the limitations of existing 3D convolution-based models (CVM-Conv3D) that are too slow and memory-intensive for mobile deployment.
• Design a flexible, configurable network architecture that allows users to balance model size, accuracy, and inference speed for different hardware platforms.
• Achieve state-of-the-art performance on both synthetic (SceneFlow) and real-world datasets while maintaining low latency and memory usage.
• Enable efficient deployment on mobile devices by avoiding memory-heavy 3D convolutions and leveraging optimized 2D-based design with configurable channel scaling.

Proposed method

• Propose a novel 2D-convolution-based network architecture (FADNet++) that replaces 3D convolutions with point-wise correlation layers to reduce computational cost.
• Integrate residual blocks and multi-scale residual learning to enhance feature representation and model accuracy without increasing inference latency.
• Introduce a configurable channel scaling ratio to generate multiple model variants (FADNet-T, FADNet-S, FADNet-M, FADNet++) with varying parameter counts and inference speeds.
• Train all variants using a unified, scalable training strategy that boosts accuracy across all configurations.
• Leverage efficient 2D operations and optimized layer implementations (e.g., cuDNN) to ensure fast inference on both server and mobile GPUs.
• Use a cost volume-like representation via point-wise correlation to capture cross-image feature relationships without the computational burden of 3D convolutions.

Experimental results

Research questions

• RQ1 Can a 2D-convolution-based network achieve state-of-the-art disparity estimation accuracy while maintaining real-time inference speed on both server and mobile GPUs?
• RQ2 How effective is the proposed configurable channel scaling strategy in balancing model accuracy and inference speed across different hardware platforms?
• RQ3 Can the FADNet++ framework outperform existing 3D-convolution-based models (CVM-Conv3D) in both accuracy and speed, especially on memory-constrained mobile devices?
• RQ4 To what extent does the use of point-wise correlation layers reduce computational cost compared to 3D convolutions while preserving feature learning capability?
• RQ5 Can the model achieve sub-100ms inference time (i.e., >15 FPS) on mobile GPUs like Jetson AGX while maintaining low end-to-end error (EPE) compared to existing real-time methods?

Key findings

• On the SceneFlow dataset, FADNet++ achieves an EPE of 0.76 with a runtime of 0.053 seconds on an RTX 2070, outperforming 3D-based models in speed while maintaining competitive accuracy.
• On mobile GPUs (Jetson AGX), FADNet++ runs at 15 FPS with an EPE of 0.258, achieving nearly three times better accuracy than AnyNet while maintaining similar inference speed.
• FADNet-S runs 4× and 5.9× faster than StereoNet on TX2 and AGX GPUs, respectively, while achieving comparable EPE (1.19 vs. 1.2), demonstrating superior efficiency.
• FADNet++ consumes only 2.3 GB GPU memory on V100, significantly less than 3D models like GANet (7.5 GB), enabling deployment on mobile platforms where 3D models fail due to memory constraints.
• The configurable framework allows users to select model variants (FADNet-T to FADNet++) that achieve EPEs between 0.76 and 1.83 while maintaining inference speeds from 0.013s to 0.053s across different GPUs.
• On the Jetson AGX mobile GPU, FADNet-T achieves 15 FPS with an EPE of 0.043, setting a new SOTA for real-time disparity estimation on mobile devices under the 15 FPS constraint.