[Paper Review] PerMO: Perceiving More at Once from a Single Image for Autonomous Driving

Feixiang Lu, Zongdai Liu+9 more|arXiv (Cornell University)|Jul 16, 2020

Advanced Neural Network Applications61 references4 citations

TL;DR

PerMO proposes an end-to-end deep learning framework that enables high-accuracy 3D vehicle perception from a single image by combining part-based deformable models with dense 2D/3D correspondence prediction. It achieves state-of-the-art performance in 2D instance segmentation (+4.4 mAP), 6-DoF pose estimation (+9.11 points), and 3D detection (+1.37 mAP), with near-interactive inference speeds on commodity GPUs and a publicly released dataset and codebase.

ABSTRACT

We present a novel approach to detect, segment, and reconstruct complete textured 3D models of vehicles from a single image for autonomous driving. Our approach combines the strengths of deep learning and the elegance of traditional techniques from part-based deformable model representation to produce high-quality 3D models in the presence of severe occlusions. We present a new part-based deformable vehicle model that is used for instance segmentation and automatically generate a dataset that contains dense correspondences between 2D images and 3D models. We also present a novel end-to-end deep neural network to predict dense 2D/3D mapping and highlight its benefits. Based on the dense mapping, we are able to compute precise 6-DoF poses and 3D reconstruction results at almost interactive rates on a commodity GPU. We have integrated these algorithms with an autonomous driving system. In practice, our method outperforms the state-of-the-art methods for all major vehicle parsing tasks: 2D instance segmentation by 4.4 points (mAP), 6-DoF pose estimation by 9.11 points, and 3D detection by 1.37. Moreover, we have released all of the source code, dataset, and the trained model on Github.

Motivation & Objective

• To address the challenge of robust 3D vehicle perception under severe occlusions in autonomous driving scenarios using only a single image.
• To develop a part-based deformable vehicle model that enhances robustness to partial occlusions and enables precise 3D reconstruction.
• To generate a large-scale, dense 2D/3D correspondence dataset for training and benchmarking 3D perception models.
• To enable end-to-end prediction of 2D/3D mappings for instance segmentation, 6-DoF pose estimation, and textured 3D reconstruction at near-interactive speeds.
• To integrate the method into autonomous driving systems and simulators for practical deployment and data-driven simulation.

Proposed method

• The method introduces a novel part-based deformable vehicle model that decomposes vehicles into semantic parts (e.g., body, tires) to improve occlusion robustness.
• It formulates a two-stage dense 2D/3D correspondence prediction: first identifying part-level regions, then predicting per-pixel correspondences within each part.
• A deep neural network is trained end-to-end to predict dense 2D-to-3D mappings, leveraging geometric priors and symmetry for accurate reconstruction.
• The framework uses a bidirectional deformation strategy—ARAP for body parts and ICP for tires—to align template CAD models to real vehicles, enabling a generalized PCA-based shape model.
• 3D reconstruction is performed using the predicted correspondences, followed by texture mapping and model refinement to produce metrically accurate, textured 3D models.
• The system is optimized for real-time inference, achieving almost interactive rates on a commodity GPU, with runtime linear in the number of detected vehicles.

Experimental results

Research questions

• RQ1 Can a part-based deformable model improve 3D vehicle perception robustness under severe occlusion in single-image scenarios?
• RQ2 How effectively can end-to-end deep learning predict dense 2D/3D correspondences for accurate 3D reconstruction from a single image?
• RQ3 To what extent does the proposed method outperform state-of-the-art approaches in 2D instance segmentation, 6-DoF pose estimation, and 3D detection?
• RQ4 Can a large-scale, dense 2D/3D correspondence dataset significantly improve the performance of 3D perception models in autonomous driving?
• RQ5 How scalable and efficient is the method in real-world autonomous driving systems and simulators?

Key findings

• PerMO achieves a 4.4-point improvement in mAP over state-of-the-art methods for 2D instance segmentation, demonstrating superior instance-level recognition and localization.
• The method improves 6-DoF pose estimation by 9.11 points in the A3DP-Abs metric, indicating significantly more accurate 3D object pose estimation under complex occlusions.
• For 3D vehicle detection, PerMO achieves a 1.37-point gain in mAP, showing enhanced accuracy in 3D bounding box prediction from single images.
• The framework enables near-interactive inference performance on a commodity GPU, with runtime scaling linearly with the number of vehicles in the scene.
• The proposed dataset, including dense 2D/3D correspondences and part-level annotations, has been released publicly and is already being used for autonomous driving simulation and behavior reasoning.
• The integration of PerMO into autonomous driving systems and simulators demonstrates its practical utility and robustness in real-world driving scenarios.