[Paper Review] Calibrating LiDAR and Camera using Semantic Mutual information

Peng Jiang, Philip R. Osteen+1 more|arXiv (Cornell University)|Apr 24, 2021

Robotics and Sensor-Based Localization32 references4 citations

TL;DR

This paper proposes a targetless, automatic LiDAR-camera extrinsic calibration method that maximizes semantic mutual information between sensor modalities using a differentiable framework based on mutual information neural estimation (MINE) and matrix exponential transformation. The method achieves state-of-the-art accuracy on KITTI360 and RELLIS-3D benchmarks, outperforming prior learning-based and optimization-based approaches by leveraging fully differentiable, semantic-aware optimization with gradient descent.

ABSTRACT

We propose an algorithm for automatic, targetless, extrinsic calibration of a LiDAR and camera system using semantic information. We achieve this goal by maximizing mutual information (MI) of semantic information between sensors, leveraging a neural network to estimate semantic mutual information, and matrix exponential for calibration computation. Using kernel-based sampling to sample data from camera measurement based on LiDAR projected points, we formulate the problem as a novel differentiable objective function which supports the use of gradient-based optimization methods. We also introduce an initial calibration method using 2D MI-based image registration. Finally, we demonstrate the robustness of our method and quantitatively analyze the accuracy on a synthetic dataset and also evaluate our algorithm qualitatively on KITTI360 and RELLIS-3D benchmark datasets, showing improvement over recent comparable approaches.

Motivation & Objective

• To develop a targetless, automatic extrinsic calibration method for LiDAR and camera systems without requiring calibration targets or precise initial guesses.
• To improve calibration accuracy by leveraging semantic information from both sensors, rather than relying on low-level features like edges or gradients.
• To enable end-to-end differentiable optimization of the calibration process using mutual information as a differentiable objective function.
• To demonstrate robustness and accuracy on synthetic and real-world datasets, including KITTI360 and RELLIS-3D, under varying label noise and data quantity.
• To provide a framework compatible with deep learning libraries for integration into broader sensor fusion pipelines.

Proposed method

• The method uses mutual information (MI) between semantic labels from LiDAR point clouds and camera images as the optimization objective, replacing traditional geometric or intensity-based metrics.
• Semantic mutual information is estimated using MINE (Mutual Information Neural Estimator), enabling differentiable MI estimation for gradient-based optimization.
• A kernel-based sampling module samples image pixels corresponding to LiDAR-projected points, aligning spatially projected 3D points with 2D image regions for joint semantic comparison.
• The transformation matrix is parameterized using matrix exponential of skew-symmetric matrices, ensuring proper SO(3) and SE(3) group constraints during optimization.
• An initial calibration is performed using 2D MI-based image registration between projected LiDAR labels and image labels to provide a robust starting point.
• The entire pipeline is differentiable and optimized via stochastic gradient descent using PyTorch or similar deep learning frameworks.

Experimental results

Research questions

• RQ1 Can semantic mutual information serve as a robust, differentiable objective for targetless LiDAR-camera calibration?
• RQ2 How does the proposed differentiable framework based on MINE and matrix exponential improve calibration accuracy compared to non-differentiable or non-semantic methods?
• RQ3 To what extent does label noise or data quantity affect the performance of the semantic MI-based calibration?
• RQ4 Can the method achieve competitive calibration accuracy on real-world benchmarks like KITTI360 and RELLIS-3D without ground-truth initialization?
• RQ5 How does the method compare to state-of-the-art approaches such as SOIC and PMI in terms of alignment quality and robustness?

Key findings

• On the synthetic Carla dataset, calibration error variance decreased with increasing frame count, showing improved MI estimation and convergence stability with more data.
• The method achieved a roll of 73.98°, pitch of -71.18°, yaw of 64.57°, and translation of (0.02, -0.14, -1.36) m on KITTI360, closely matching the provided dataset parameters.
• On RELLIS-3D, the method achieved a roll of 70.24°, pitch of 67.63°, yaw of -67.32°, and translation of (-0.05, -0.19, -0.06) m, outperforming SOIC and PMI in alignment quality.
• With 20% label noise, the method showed increased error variance but maintained reasonable accuracy, demonstrating robustness to moderate semantic label errors.
• Visual comparisons on KITTI360 and RELLIS-3D showed superior cross-sensor alignment with the proposed method compared to SOIC and PMI, especially in complex urban scenes.
• The method achieved state-of-the-art performance on both benchmarks, with calibration results closest to the provided dataset parameters, validating its effectiveness and robustness.