[Paper Review] CalibNet: Self-Supervised Extrinsic Calibration using 3D Spatial Transformer Networks.

Ganesh Iyer, Ranjith Ram+2 more|arXiv (Cornell University)|Mar 22, 2018

Robotics and Sensor-Based LocalizationEngineering17 references16 citations

TL;DR

CalibNet is a self-supervised deep learning method that estimates 6-DoF extrinsic calibration between 3D LiDARs and 2D cameras in real-time without requiring calibration targets or direct supervision. It learns by optimizing geometric and photometric consistency between point clouds and images, achieving accurate calibration across diverse mis-calibration scenarios without retraining.

ABSTRACT

3D LiDARs and 2D cameras are increasingly being used alongside each other in sensor rigs for perception tasks. Before these sensors can be used to gather meaningful data, however, their extrinsics (and intrinsics) need to be accurately calibrated, as the performance of the sensor rig is extremely sensitive to these calibration parameters. A vast majority of existing calibration techniques require significant amounts of data and/or calibration targets and human effort, severely impacting their applicability in large-scale production systems. We address this gap with CalibNet: a self-supervised deep network capable of automatically estimating the 6-DoF rigid body transformation between a 3D LiDAR and a 2D camera in real-time. CalibNet alleviates the need for calibration targets, thereby resulting in significant savings in calibration efforts. During training, the network only takes as input a LiDAR point cloud, the corresponding monocular image, and the camera calibration matrix K. At train time, we do not impose direct supervision (i.e., we do not directly regress to the calibration parameters, for example). Instead, we train the network to predict calibration parameters that maximize the geometric and photometric consistency of the input images and point clouds. CalibNet learns to iteratively solve the underlying geometric problem and accurately predicts extrinsic calibration parameters for a wide range of mis-calibrations, without requiring retraining or domain adaptation. The project page is hosted at this https URL

Motivation & Objective

• To eliminate the need for calibration targets and human-annotated ground-truth in LiDAR-camera extrinsic calibration.
• To enable real-time, self-supervised estimation of 6-DoF rigid body transformations between 3D LiDARs and 2D cameras.
• To reduce calibration effort in large-scale deployment of sensor rigs by eliminating manual calibration procedures.
• To develop a method that generalizes across diverse mis-calibration scenarios without retraining or domain adaptation.
• To train a network using only raw input data—LiDAR point clouds, monocular images, and intrinsic camera matrix K—without direct supervision on calibration parameters.

Proposed method

• The network uses a 3D spatial transformer to predict a 6-DoF transformation that aligns the LiDAR point cloud with the camera image in a differentiable manner.
• It trains via a loss function that maximizes geometric consistency between projected 3D points and image features, and photometric consistency between the transformed point cloud projection and the input image.
• The network is trained end-to-end using only the input LiDAR point cloud, corresponding image, and camera intrinsic matrix K, with no ground-truth calibration labels.
• The method employs iterative optimization through backpropagation to refine the predicted transformation based on consistency losses.
• The spatial transformer enables differentiable spatial transformation of the 3D point cloud to align with the 2D image space, enabling gradient-based learning.
• The loss function combines both geometric reprojection error and photometric reconstruction error to guide the network toward accurate calibration.

Experimental results

Research questions

• RQ1 Can a deep learning model estimate LiDAR-camera extrinsic calibration without any ground-truth labels or calibration targets?
• RQ2 Can the model generalize to a wide range of initial mis-calibrations without retraining or domain adaptation?
• RQ3 Does self-supervised training based on geometric and photometric consistency yield accurate and robust calibration results?
• RQ4 Can the method achieve real-time inference for deployment in large-scale sensor systems?
• RQ5 How does the performance of the self-supervised approach compare to supervised or target-based calibration methods in terms of accuracy and efficiency?

Key findings

• CalibNet achieves accurate 6-DoF extrinsic calibration without requiring any calibration targets or human-annotated ground-truth data.
• The method generalizes across diverse initial mis-calibration states without retraining or domain adaptation.
• The network learns to predict calibration parameters that maximize both geometric and photometric consistency between LiDAR point clouds and camera images.
• CalibNet enables real-time inference, making it suitable for large-scale deployment in production sensor systems.
• The self-supervised training paradigm significantly reduces calibration effort compared to traditional target-based or supervised methods.
• The approach demonstrates robustness to initial calibration errors, converging to accurate solutions through consistency-based optimization.