[Paper Review] Pixel-level Extrinsic Self Calibration of High Resolution LiDAR and Camera in Targetless Environments

Chongjian Yuan, Xiyuan Liu+2 more|arXiv (Cornell University)|Mar 2, 2021

Robotics and Sensor-Based LocalizationEngineering23 references19 citations

TL;DR

This paper proposes a targetless, pixel-level extrinsic calibration method for high-resolution LiDAR and camera systems using natural edge features. By leveraging a novel LiDAR edge extraction technique based on voxel cutting and plane fitting, the method achieves pixel-level accuracy without relying on checkerboards or artificial targets, demonstrating robustness and consistency across diverse indoor and outdoor scenes.

ABSTRACT

In this letter, we present a novel method for automatic extrinsic calibration of high-resolution LiDARs and RGB cameras in targetless environments. Our approach does not require checkerboards but can achieve pixel-level accuracy by aligning natural edge features in the two sensors. On the theory level, we analyze the constraints imposed by edge features and the sensitivity of calibration accuracy with respect to edge distribution in the scene. On the implementation level, we carefully investigate the physical measuring principles of LiDARs and propose an efficient and accurate LiDAR edge extraction method based on point cloud voxel cutting and plane fitting. Due to the edges' richness in natural scenes, we have carried out experiments in many indoor and outdoor scenes. The results show that this method has high robustness, accuracy, and consistency. It can promote the research and application of the fusion between LiDAR and camera. We have open-sourced our code on GitHub to benefit the community.

Motivation & Objective

• Address the limitations of target-based calibration methods, which require external markers like checkerboards and are impractical for dynamic or long-range applications.
• Overcome the instability and inaccuracies introduced by depth-discontinuous edges in high-resolution LiDAR point clouds, especially from solid-state sensors like Livox.
• Develop a robust, automatic, and targetless calibration method suitable for both spinning and solid-state LiDARs in real-world environments.
• Ensure high accuracy and consistency in extrinsic calibration by analyzing edge distribution constraints and incorporating accurate LiDAR noise models.
• Enable in-field or online calibration during data collection by eliminating dependency on pre-placed calibration targets.

Proposed method

• Propose a depth-continuous edge extraction method for LiDAR by applying voxel cutting and plane fitting to improve edge reliability over depth-discontinuity-based methods.
• Use natural 2D image edges and 3D LiDAR edges as correspondences to minimize reprojection error in a nonlinear optimization framework.
• Incorporate a physically motivated LiDAR noise model to enhance calibration accuracy and convergence stability.
• Formulate the calibration problem as a nonlinear least-squares optimization that minimizes the reprojection error between 2D image edges and their 3D projections.
• Apply a robust initialization strategy using random perturbations within ±3° in rotation and ±5 cm in translation to evaluate convergence robustness.
• Validate the method on both Livox Avia and Ouster OS2-64 LiDARs, demonstrating applicability across different LiDAR types including low-resolution spinning sensors when combined with motion-based registration.

Experimental results

Research questions

• RQ1 Can natural edge features in unstructured scenes provide sufficient and reliable constraints for high-accuracy extrinsic calibration without artificial targets?
• RQ2 How does the distribution and orientation of edges in a scene affect the accuracy and stability of LiDAR-camera calibration?
• RQ3 Is a depth-continuous edge extraction method based on voxel cutting and plane fitting more accurate and reliable than depth-discontinuity-based edge detection for high-resolution LiDARs?
• RQ4 Can the proposed method achieve pixel-level calibration accuracy comparable to target-based methods in real-world, diverse environments?
• RQ5 How robust is the method to initialization uncertainty and sensor noise, especially in long-range or sparse point cloud scenarios?

Key findings

• The proposed method achieves pixel-level calibration accuracy in both indoor and outdoor scenes, matching or exceeding the performance of target-based methods like ACSC and Zhou [6].
• The method demonstrates high robustness, converging to consistent extrinsic parameters across 20 different random initializations within ±3° in rotation and ±5 cm in translation.
• Experiments on the Ouster OS2-64 LiDAR show that the method is effective even with lower-resolution scanning when combined with LiDAR-inertial odometry to enhance point cloud resolution.
• The use of depth-continuous edges extracted via voxel cutting and plane fitting leads to significantly more reliable calibration than depth-discontinuity-based edge detection.
• Quantitative comparison shows similar residual distributions between the proposed method and target-based approaches, with the proposed method achieving comparable accuracy using only one data pair versus multiple placements in target-based methods.
• The method successfully mitigates the effect of laser beam divergence, as evidenced by consistent point cloud coloring and measured inflation width (1.4 cm at 6 m) aligning with theoretical beam divergence (0.28° half-angle).