[Paper Review] R4Dyn: Exploring Radar for Self-Supervised Monocular Depth Estimation of Dynamic Scenes

Stefano Gasperini|arXiv (Cornell University)|Aug 10, 2021

Advanced Optical Sensing TechnologiesPhysics and Astronomy34 references31 citations

TL;DR

R4Dyn introduces a novel self-supervised monocular depth estimation framework that leverages low-cost automotive radar as weak supervision during training and optionally as input at inference to improve depth estimation on dynamic objects. By filtering and expanding sparse radar detections, the method reduces errors on moving vehicles by 37% on the nuScenes dataset, significantly outperforming baseline self-supervised models like Monodepth2 in dynamic scenes.

ABSTRACT

While self-supervised monocular depth estimation in driving scenarios has achieved comparable performance to supervised approaches, violations of the static world assumption can still lead to erroneous depth predictions of traffic participants, posing a potential safety issue. In this paper, we present R4Dyn, a novel set of techniques to use cost-efficient radar data on top of a self-supervised depth estimation framework. In particular, we show how radar can be used during training as weak supervision signal, as well as an extra input to enhance the estimation robustness at inference time. Since automotive radars are readily available, this allows to collect training data from a variety of existing vehicles. Moreover, by filtering and expanding the signal to make it compatible with learning-based approaches, we address radar inherent issues, such as noise and sparsity. With R4Dyn we are able to overcome a major limitation of self-supervised depth estimation, i.e. the prediction of traffic participants. We substantially improve the estimation on dynamic objects, such as cars by 37% on the challenging nuScenes dataset, hence demonstrating that radar is a valuable additional sensor for monocular depth estimation in autonomous vehicles.

Motivation & Objective

• To address the failure of self-supervised monocular depth estimation on dynamic traffic participants due to violations of the static world assumption.
• To enable training on existing vehicles by leveraging widely available automotive radar data instead of requiring LiDAR-supervised data.
• To improve robustness and accuracy of depth estimation on safety-critical dynamic objects such as oncoming vehicles and pedestrians.
• To develop a method that integrates radar both as a weak supervision signal during training and as an input modality at inference, enhancing performance without increasing model complexity.

Proposed method

• Proposes a novel weak radar loss that uses filtered and expanded radar detections as a supervision signal during training, improving depth estimation on dynamic objects.
• Introduces a signal processing pipeline to filter raw radar detections using 2D bounding boxes and expand them spatially to create dense, learnable radar features compatible with deep learning models.
• Extends a self-supervised depth estimation framework (e.g., Monodepth2) by incorporating radar data through a multi-stream encoder and early fusion, enabling very sparse depth completion at inference.
• Uses a multi-task learning setup with photometric loss, weak velocity loss, and weak radar loss to jointly optimize depth and pose estimation.
• Applies data augmentation and domain adaptation techniques to ensure generalization across datasets, including transfer from nuScenes to KITTI.
• Sub-samples LiDAR data to simulate radar sparsity and noise, enabling transfer learning experiments on KITTI without fine-tuning.

Experimental results

Research questions

• RQ1 Can radar data be effectively used as a weak supervision signal to improve self-supervised monocular depth estimation on dynamic objects?
• RQ2 How can sparse and noisy automotive radar data be preprocessed to be compatible with deep learning-based depth estimation networks?
• RQ3 Does integrating radar during training and inference improve depth estimation accuracy on safety-critical dynamic objects such as oncoming vehicles and pedestrians?
• RQ4 Can the proposed method generalize to different datasets and real-world scenarios, including those without LiDAR?

Key findings

• R4Dyn reduces depth estimation error on dynamic objects (cars) by 37% compared to Monodepth2 on the nuScenes dataset, demonstrating significant improvement in handling moving traffic.
• The method achieves a 77.86% mIoU on the 'Vehicles' class and 80.86% on 'Non-Parked Vehicles', outperforming all baseline methods in class-wise evaluation.
• On the KITTI dataset, R4Dyn generalizes well without fine-tuning, achieving an 11.5% relative error improvement over Monodepth2, despite a large domain gap.
• The weak radar loss alone (R4Dyn-L) reduces AbsRel error on cars by 30.5% compared to the baseline, showing the value of radar as supervision even without input fusion.
• R4Dyn-LI, which uses both radar supervision and input fusion, achieves the best performance with an AbsRel of 0.1551 on cars and 0.2222 on objects, outperforming LiDAR-supervised prior work on object classes.
• Qualitative results show that R4Dyn correctly estimates depth for oncoming vehicles, while Monodepth2 severely underestimates their distance, highlighting the method’s robustness in safety-critical scenarios.