[Paper Review] DEVIANT: Depth EquiVarIAnt NeTwork for Monocular 3D Object Detection

Abhinav Kumar, Garrick Brazil+3 more|arXiv (Cornell University)|Jul 21, 2022

Advanced Neural Network Applications4 citations

TL;DR

DEVIANT proposes a depth-equivariant neural network for monocular 3D object detection by replacing standard convolutions with scale-equivariant steerable blocks, enabling equivariance to depth translations in the projective manifold. This design improves depth estimation consistency, achieving state-of-the-art performance on KITTI and Waymo benchmarks in the image-only category and superior generalization in cross-dataset evaluation.

ABSTRACT

Modern neural networks use building blocks such as convolutions that are equivariant to arbitrary 2D translations. However, these vanilla blocks are not equivariant to arbitrary 3D translations in the projective manifold. Even then, all monocular 3D detectors use vanilla blocks to obtain the 3D coordinates, a task for which the vanilla blocks are not designed for. This paper takes the first step towards convolutions equivariant to arbitrary 3D translations in the projective manifold. Since the depth is the hardest to estimate for monocular detection, this paper proposes Depth EquiVarIAnt NeTwork (DEVIANT) built with existing scale equivariant steerable blocks. As a result, DEVIANT is equivariant to the depth translations in the projective manifold whereas vanilla networks are not. The additional depth equivariance forces the DEVIANT to learn consistent depth estimates, and therefore, DEVIANT achieves state-of-the-art monocular 3D detection results on KITTI and Waymo datasets in the image-only category and performs competitively to methods using extra information. Moreover, DEVIANT works better than vanilla networks in cross-dataset evaluation. Code and models at https://github.com/abhi1kumar/DEVIANT

Motivation & Objective

• To address the mismatch between standard CNNs, which are equivariant only to 2D translations, and the 3D projective manifold where monocular detection operates.
• To improve depth estimation consistency in monocular 3D detection, which is inherently ill-posed and error-prone.
• To enhance generalization, especially in cross-dataset evaluation, by embedding depth equivariance into the backbone.
• To demonstrate that equivariance to depth translations in the projective manifold leads to more robust and accurate 3D detection.

Proposed method

• Replace standard convolutional blocks in the backbone with scale-equivariant steerable convolutional blocks to enforce equivariance to depth (scale) transformations.
• Leverage theoretical results from [30] under realistic assumptions of ego-motion in autonomous driving to derive scale transformations as proxies for depth translations.
• Enforce depth equivariance by ensuring that feature maps transform consistently under scale transformations corresponding to ego-motion in depth.
• Use existing scale-equivariant building blocks to maintain computational efficiency while embedding equivariance into the network architecture.
• Train the network using standard detection heads (e.g., from GUP-Net or CenterNet) with the modified backbone to preserve detection head performance.
• Validate equivariance via controlled experiments measuring equivariance error under scale transformations.

Experimental results

Research questions

• RQ1 Can depth-equivariant convolutions improve monocular 3D object detection performance by enforcing consistency in depth estimation?
• RQ2 Does embedding equivariance to depth translations in the projective manifold lead to better generalization across datasets?
• RQ3 How does the proposed DEVIANT architecture compare to vanilla CNN backbones in terms of depth estimation accuracy and robustness?
• RQ4 To what extent does depth equivariance reduce the generalization gap between training and validation in monocular 3D detection?

Key findings

• DEVIANT achieves state-of-the-art performance on the KITTI and Waymo datasets in the image-only category, outperforming prior SOTA methods.
• The model shows significantly lower depth equivariance error compared to vanilla CNNs, confirming successful integration of depth equivariance.
• In cross-dataset evaluation (KITTI model tested on nuScenes), DEVIANT generalizes better than baseline models, detecting fewer false positives and producing more accurate 3D boxes.
• Qualitative results on KITTI, nuScenes, and Waymo show that DEVIANT predictions are consistently closer to ground truth than GUP-Net, with more stable box orientations even without temporal information.
• The equivariance error demo confirms that DEVIANT maintains lower error across scale transformations, validating the theoretical design.
• The model achieves stable and accurate 3D detection on raw KITTI video sequences without using temporal information, demonstrating robustness.