[Paper Review] DeepI2P: Image-to-Point Cloud Registration via Deep Classification

Jiaxin Li, Gim Hee Lee|arXiv (Cornell University)|Apr 8, 2021

Robotics and Sensor-Based Localization42 references4 citations

TL;DR

DeepI2P proposes a novel image-to-point cloud registration method that bypasses feature matching by framing registration as a two-stage problem: first classifying which 3D points lie within the camera frustum using a deep neural network, then optimizing the camera pose via inverse projection. The approach achieves state-of-the-art performance on Oxford Robotcar and KITTI datasets with a 1.65m RTE and 4.14° RRE, demonstrating feasibility without explicit cross-modal feature descriptors.

ABSTRACT

This paper presents DeepI2P: a novel approach for cross-modality registration between an image and a point cloud. Given an image (e.g. from a rgb-camera) and a general point cloud (e.g. from a 3D Lidar scanner) captured at different locations in the same scene, our method estimates the relative rigid transformation between the coordinate frames of the camera and Lidar. Learning common feature descriptors to establish correspondences for the registration is inherently challenging due to the lack of appearance and geometric correlations across the two modalities. We circumvent the difficulty by converting the registration problem into a classification and inverse camera projection optimization problem. A classification neural network is designed to label whether the projection of each point in the point cloud is within or beyond the camera frustum. These labeled points are subsequently passed into a novel inverse camera projection solver to estimate the relative pose. Extensive experimental results on Oxford Robotcar and KITTI datasets demonstrate the feasibility of our approach. Our source code is available at https://github.com/lijx10/DeepI2P

Motivation & Objective

• To address the challenge of cross-modality image-to-point cloud registration, which is difficult due to drastic differences in appearance and geometry between modalities.
• To eliminate the need for learning cross-modal feature descriptors, which are prone to failure due to modality mismatch.
• To reduce memory overhead by avoiding storage of high-dimensional feature descriptors, enabling efficient deployment on mobile and robotic platforms.
• To develop a robust, end-to-end framework that leverages deep learning for frustum classification and optimization for pose estimation.

Proposed method

• A two-branch convolutional neural network with cross-attention modules is used to classify each 3D point as inside or outside the camera frustum based on image and point cloud inputs.
• The classification network outputs a binary label per 3D point indicating whether its projection lies within the image's field of view.
• The pose estimation stage formulates the problem as an unconstrained continuous optimization to minimize the reprojection error of in-frustum points.
• The optimization uses Gauss-Newton algorithm to solve for the rigid transformation (rotation and translation) that best aligns the predicted in-frustum points with the image plane.
• The method avoids feature matching entirely, relying instead on geometric consistency through inverse camera projection.
• The framework is trained and evaluated on Oxford Robotcar and KITTI datasets, with ablation studies on initialization, attention, and point density.

Figure 2: Our network architecture for the classification problem.

Experimental results

Research questions

• RQ1 Can cross-modality image-to-point cloud registration be achieved without explicit feature matching between 2D and 3D modalities?
• RQ2 How effective is a deep learning-based frustum classification approach in replacing traditional feature-based correspondence matching?
• RQ3 To what extent does the inclusion of cross-attention modules improve classification and registration accuracy?
• RQ4 How robust is the method to variations in point cloud density and occlusion?
• RQ5 Can the inverse camera projection optimization reliably recover the camera pose from binary frustum classification outputs?

Key findings

• DeepI2P achieves a translation error (RTE) of 1.65m and rotation error (RRE) of 4.14° on the Oxford Robotcar dataset, demonstrating strong registration performance.
• On the KITTI dataset, the method achieves RTE of 1.94m and RRE of 4.63° at 5120 points, showing reasonable performance even at reduced point density.
• The ablation study confirms that removing the cross-attention module causes a significant drop in classification accuracy (from 98% to 80%) and registration error (RTE increases to 6.88m).
• The method is robust to initialization, with RTE increasing only slightly from 1.65m to 3.52m when the number of initializations is reduced from 60 to 1.
• The performance degrades with lower point density, but the drop is moderate—RTE increases from 1.65m to 1.94m when density is reduced to 1/4 of the original (20480 to 5120 points).
• Compared to Monodepth2+ICP, DeepI2P shows significantly better performance (RTE: 1.65m vs. 8.45m), especially under poor initialization, highlighting its robustness.

Figure 3: Visualizations of the Gauss-Newton at iteration 0 / 40 / 80 from left to right. Green points are classified as inside image FoV.