[Paper Review] Probabilistic and Geometric Depth: Detecting Objects in Perspective

Tai Wang, Xinge Zhu+2 more|arXiv (Cornell University)|Jul 29, 2021

Advanced Neural Network ApplicationsComputer Science50 references94 citations

TL;DR

The paper introduces PGD, which combines probabilistic depth uncertainty with a geometric depth propagation graph to improve monocular 3D object detection, achieving state-of-the-art results on KITTI and nuScenes while remaining real-time.

ABSTRACT

3D object detection is an important capability needed in various practical applications such as driver assistance systems. Monocular 3D detection, as a representative general setting among image-based approaches, provides a more economical solution than conventional settings relying on LiDARs but still yields unsatisfactory results. This paper first presents a systematic study on this problem. We observe that the current monocular 3D detection can be simplified as an instance depth estimation problem: The inaccurate instance depth blocks all the other 3D attribute predictions from improving the overall detection performance. Moreover, recent methods directly estimate the depth based on isolated instances or pixels while ignoring the geometric relations across different objects. To this end, we construct geometric relation graphs across predicted objects and use the graph to facilitate depth estimation. As the preliminary depth estimation of each instance is usually inaccurate in this ill-posed setting, we incorporate a probabilistic representation to capture the uncertainty. It provides an important indicator to identify confident predictions and further guide the depth propagation. Despite the simplicity of the basic idea, our method, PGD, obtains significant improvements on KITTI and nuScenes benchmarks, achieving 1st place out of all monocular vision-only methods while still maintaining real-time efficiency. Code and models will be released at https://github.com/open-mmlab/mmdetection3d.

Motivation & Objective

• Show that instance depth estimation is the critical bottleneck in monocular 3D detection and that improving depth estimation boosts overall performance.
• Propose a probabilistic depth representation to capture uncertainty for each object depth.
• Introduce a depth propagation graph based on perspective geometry to leverage contextual relations among objects.
• Fuse probabilistic local depth with graph-based geometric depth through a learned fusion mechanism.
• Demonstrate real-time performance and state-of-the-art results on KITTI and nuScenes using an end-to-end framework.

Proposed method

• Model depth as a probabilistic distribution over depth intervals and compute an expected depth D_P.
• Represent depth uncertainty via the top-k confidence scores from the depth distribution and derive a depth score s^d.
• Construct a depth propagation graph using perspective geometry to relate depths across objects, with edge pruning and gating based on depth confidence, 2D proximity, and class similarity.
• Compute a geometry-based depth D_G as a weighted sum of other objects' depth predictions using edge scores, ensuring a gradient-free propagation stage.
• Fuse local depth D_L (combination of direct regression D_R and probabilistic depth D_P) with geometric depth D_G through a location-aware map alpha: D = sigma(alpha) * D_L + (1 - sigma(alpha)) * D_G.
• Integrate the final depth prediction into an end-to-end FCOS3D-based detector and train with standard loss functions.

Experimental results

Research questions

• RQ1 Can the dominant bottleneck in monocular 3D detection be reframed as instance depth estimation, and does addressing it improve all 3D predictions?
• RQ2 Does a probabilistic depth representation improve depth estimation by capturing uncertainty and guiding propagation?
• RQ3 Do geometric relations among multiple objects, via a depth propagation graph, provide significant benefits for depth estimation beyond isolated instances?
• RQ4 Can a simple fusion of probabilistic/local and geometric depths yield robust, real-time monocular 3D detection performance?
• RQ5 How does the proposed PGD approach perform on standard benchmarks (KITTI, nuScenes) compared to prior monocular methods?

Key findings

• Depth is identified as the main bottleneck; accurate depth enables improvements in other 3D attributes.
• Probabilistic depth estimation (D_P) with an uncertainty score improves depth accuracy over direct regression alone.
• Depth propagation through a perspective-geometry graph (D_G) leverages contextual cues from neighboring objects to refine depths.
• Fusion with a location-aware map (D_L) and gating mechanisms leads to significant performance gains without sacrificing speed.
• PGD achieves first place among monocular vision-only methods on KITTI and nuScenes benchmarks while running in real-time (≈36 Hz).
• A simple FCOS3D-based backbone suffices when augmented with PGD components, demonstrating the effectiveness of depth-focused design for perspective-based detection.