[Paper Review] Depth Estimation from Monocular Images and Sparse radar using Deep Ordinal Regression Network

Chen-Chou Lo, Patrick Vandewalle|arXiv (Cornell University)|Jul 15, 2021

Advanced Vision and Imaging19 references33 citations

TL;DR

This paper proposes a late fusion method that integrates height-extended, multi-frame sparse radar data with monocular RGB images using a deep ordinal regression network (DORN) to improve dense depth estimation. By extending radar points vertically from 0.5m to 0.25–2m and applying a two-stage filtering strategy, the method increases data density and reduces error, achieving state-of-the-art performance on the nuScenes dataset across day, night, and rainy conditions.

ABSTRACT

We integrate sparse radar data into a monocular depth estimation model and introduce a novel preprocessing method for reducing the sparseness and limited field of view provided by radar. We explore the intrinsic error of different radar modalities and show our proposed method results in more data points with reduced error. We further propose a novel method for estimating dense depth maps from monocular 2D images and sparse radar measurements using deep learning based on the deep ordinal regression network by Fu et al. Radar data are integrated by first converting the sparse 2D points to a height-extended 3D measurement and then including it into the network using a late fusion approach. Experiments are conducted on the nuScenes dataset. Our experiments demonstrate state-of-the-art performance in both day and night scenes.

Motivation & Objective

• To address the limitations of monocular depth estimation, which lacks absolute depth and suffers from ill-posed reconstruction, by integrating sparse radar data.
• To overcome radar's inherent sparseness, noise, and limited vertical field of view in depth estimation tasks.
• To demonstrate that even monocular depth models not designed for multi-modal input can benefit from properly preprocessed radar data via late fusion.
• To improve depth estimation accuracy in challenging conditions such as night and rain using radar as a complementary modality.

Proposed method

• Radar data from multiple frames are combined and extended vertically from a single height (0.5m) to a range of 0.25m to 2m to increase spatial coverage and data density.
• A two-stage filtering strategy is applied to remove noisy radar points, improving the quality of the height-extended radar measurements.
• The preprocessed height-extended radar is fused with RGB images using a late fusion approach, where features from RGB and radar are concatenated after the encoder phase in the DORN network.
• The DORN model uses ordinal regression loss with spacing-increasing discretization over a depth range of 1–80m, using 80 ordinal classes.
• Data augmentation including gamma contrast, brightness, color adjustment, and horizontal flipping is applied during training to improve robustness.
• Ground truth depth is generated via interpolation from sparse lidar and RGB using a colorization method, while evaluation is performed using sparse lidar annotations.

Experimental results

Research questions

• RQ1 Can height extension of sparse radar points improve depth estimation accuracy despite introducing potential noise?
• RQ2 Does late fusion of preprocessed radar data with monocular RGB images yield better performance than early fusion or RGB-only baselines?
• RQ3 How does the integration of radar data affect depth estimation performance in low-visibility conditions such as night and rain?
• RQ4 Can a monocular depth model benefit from radar input even when not explicitly designed for multi-modal fusion?

Key findings

• Height extension of radar points increases the average number of points from 215 to 4,510, reducing intrinsic error: δ1 improved from 0.350 to 0.187 and RMSE from 25.536 to 12.184.
• The late fusion method with height-extended radar achieved the best performance, with δ1 = 0.887, RMSE = 5.194, and AbsRel = 0.107 on the overall test set.
• The proposed method outperformed the RGB-only baseline across all conditions, with the largest performance gain observed in night scenes (δ1 improved from 0.764 to 0.782).
• Applying a self-adaptive δ2 filter to the height-extended radar further improved results, achieving δ1 = 0.892, RMSE = 5.082, and AbsRel = 0.107, especially in rainy conditions.
• Visual comparisons show that the proposed model produces more detailed depth maps than the RGB-only baseline, particularly in night and rainy scenes.
• The method demonstrates robustness to adverse weather and lighting, with consistent improvements in both day and night scenarios.