[Paper Review] LiDAR-Camera Calibration using 3D-3D Point correspondences

Ankit Dhall, Kunal Chelani+2 more|arXiv (Cornell University)|May 27, 2017

Robotics and Sensor-Based LocalizationEngineering1 references148 citations

TL;DR

The paper presents a closed-form extrinsic calibration method for LiDAR–camera using 3D-3D point correspondences, demonstrated with ArUco/cardboard markers, ICP/Kabsch comparisons, and multi-camera fusion.

ABSTRACT

With the advent of autonomous vehicles, LiDAR and cameras have become an indispensable combination of sensors. They both provide rich and complementary data which can be used by various algorithms and machine learning to sense and make vital inferences about the surroundings. We propose a novel pipeline and experimental setup to find accurate rigid-body transformation for extrinsically calibrating a LiDAR and a camera. The pipeling uses 3D-3D point correspondences in LiDAR and camera frame and gives a closed form solution. We further show the accuracy of the estimate by fusing point clouds from two stereo cameras which align perfectly with the rotation and translation estimated by our method, confirming the accuracy of our method's estimates both mathematically and visually. Taking our idea of extrinsic LiDAR-camera calibration forward, we demonstrate how two cameras with no overlapping field-of-view can also be calibrated extrinsically using 3D point correspondences. The code has been made available as open-source software in the form of a ROS package, more information about which can be sought here: https://github.com/ankitdhall/lidar_camera_calibration .

Motivation & Objective

• Provide a repeatable, accurate extrinsic calibration between a camera and a LiDAR using 3D-3D correspondences.
• Evaluate the limitations of 2D-3D calibration approaches in low-density LiDAR setups.
• Demonstrate the effectiveness of a closed-form 3D-3D solution (Kabsch/SVD) for [R|t] estimation.
• Show the utility of the method for fusing point clouds from multiple cameras, including non-overlapping FOVs.

Proposed method

• Extract known 3D correspondences in both the camera and LiDAR frames using planar markers (cardboard) and ArUco markers.
• Estimate the rigid-body transform [R|t] between sensor frames via a closed-form solution (Kabsch/SVD) with known correspondences.
• Compare against 2D-3D PnP approaches (PnP, EPnP, RaNSaC-PnP) and ICP, highlighting cases where 3D-3D outperforms 2D-3D methods.
• Incorporate multiple scans to average rotation and translation to reduce noise (rotation via quaternion averaging, translation via mean).
• Demonstrate fusion of point clouds from multiple cameras by chaining transforms through a LiDAR frame to camera frames, enabling extrinsic calibration across non-overlapping views.

Experimental results

Research questions

• RQ1 Can 3D-3D point correspondences yield accurate LiDAR-camera extrinsics in a closed-form solution?
• RQ2 Do 3D-3D methods outperform 2D-3D PnP approaches in accuracy and robustness for low-density LiDARs?
• RQ3 Can the method calibrate multiple cameras with non-overlapping fields of view using a common LiDAR as reference?
• RQ4 Does averaging across multiple scans improve the stability and accuracy of the estimated [R|t]?

Key findings

• The closed-form 3D-3D approach (Kabsch/SVD) yields rotation-translation estimates close to tape measurements with RMSE in the range ~0.024–0.026 m across datasets.
• 2D-3D methods (PnP/EPnP with RaNSaC) can produce lower back-projection error but may deviate from manual tape-based measurements, indicating that minimizing projection error alone is not sufficient for true extrinsics.
• Averaging rotations via quaternions and translations across multiple runs reduces noise, improving the robustness of the [R|t] estimate.
• The approach supports extrinsic calibration of cameras with non-overlapping FOV by fusing point clouds through a LiDAR reference, enabling multi-camera 3D fusion.
• The authors provide an open-source ROS package implementing lidar_camera_calibration for practical use.