[Paper Review] DSGN++: Exploiting Visual-Spatial Relation for Stereo-based 3D Detectors

Yilun Chen, Shijia Huang+3 more|arXiv (Cornell University)|Apr 6, 2022

Advanced Vision and Imaging4 citations

TL;DR

DSGN++ proposes a stereo-based 3D object detection framework that enhances 2D-to-3D feature transformation through depth-wise plane sweeping (DPS), dual-view stereo volumes (DSV) for multi-view feature fusion, and Stereo-LiDAR Copy-Paste for data augmentation. It achieves state-of-the-art performance on the KITTI benchmark, with 69.12 AP 3D for cars, outperforming prior camera-only detectors without additional bells and whistles.

ABSTRACT

Camera-based 3D object detectors are welcome due to their wider deployment and lower price than LiDAR sensors. We first revisit the prior stereo detector DSGN for its stereo volume construction ways for representing both 3D geometry and semantics. We polish the stereo modeling and propose the advanced version, DSGN++, aiming to enhance effective information flow throughout the 2D-to-3D pipeline in three main aspects. First, to effectively lift the 2D information to stereo volume, we propose depth-wise plane sweeping (DPS) that allows denser connections and extracts depth-guided features. Second, for grasping differently spaced features, we present a novel stereo volume -- Dual-view Stereo Volume (DSV) that integrates front-view and top-view features and reconstructs sub-voxel depth in the camera frustum. Third, as the foreground region becomes less dominant in 3D space, we propose a multi-modal data editing strategy -- Stereo-LiDAR Copy-Paste, which ensures cross-modal alignment and improves data efficiency. Without bells and whistles, extensive experiments in various modality setups on the popular KITTI benchmark show that our method consistently outperforms other camera-based 3D detectors for all categories. Code is available at https://github.com/chenyilun95/DSGN2.

Motivation & Objective

• To address the limited information flow in 2D-to-3D feature lifting for stereo-based 3D detectors.
• To improve 3D representation by fusing front-view and top-view features for irregularly shaped objects like pedestrians and cyclists.
• To mitigate data imbalance and improve generalization in camera-based 3D detection through a novel cross-modal data editing strategy.
• To enhance model efficiency and robustness in long-range and occluded scenarios without relying on LiDAR.

Proposed method

• Depth-wise plane sweeping (DPS) enables denser, depth-guided feature propagation by expanding channel capacity and using cyclic slicing for local continuity in 3D volume construction.
• Dual-view Stereo Volume (DSV) integrates features from front-view plane-sweep volumes and top-view 3D-geometry volumes to better capture differently shaped objects.
• Stereo-LiDAR Copy-Paste performs data editing by copying LiDAR-generated 3D objects into stereo images, ensuring cross-modal alignment and improving data efficiency.
• The framework uses a two-stream 2D backbone (e.g., ResNet-34) to extract features, followed by stereo volume construction and a BEV-based 3D detector for final prediction.
• Cyclic slicing in DPS ensures smooth feature transitions across depth planes, improving resolution and continuity in the 3D volume.
• The method is end-to-end trainable and does not require additional head designs or complex post-processing.

Experimental results

Research questions

• RQ1 How can 2D-to-3D feature lifting be improved to preserve richer semantic and geometric information?
• RQ2 Can combining front-view and top-view stereo volumes yield better 3D representations for irregularly shaped objects?
• RQ3 To what extent can cross-modal data augmentation improve data efficiency and generalization in camera-based 3D detection?
• RQ4 How does the proposed stereo modeling affect performance on long-range and occluded object detection?

Key findings

• DSGN++ achieves 69.12 AP 3D for cars on the KITTI benchmark, outperforming prior stereo-based detectors without additional components.
• The Stereo-LiDAR Copy-Paste strategy improves AP 3D by +2.60 for cars and +2.53 for cars when increasing positive sample count, demonstrating strong data efficiency gains.
• The use of DSV reduces foreground depth estimation error from 0.63m to 0.57m, indicating improved geometric accuracy across distances.
• The efficient R18-DSGN++ variant achieves 68.12 AP 3D with half the parameters of the full model, showing strong speed-accuracy trade-offs.
• Inference time is reduced to 0.178s with ResNet-18, demonstrating that the 2D backbone is underutilized in standard pipelines.
• Qualitative results show robust detection of occluded and distant objects, with accurate 3D bounding boxes even at 50m range.