[Paper Review] TartanCalib: Iterative Wide-Angle Lens Calibration using Adaptive SubPixel Refinement of AprilTags

Bardienus P Duisterhof, Yaoyu Hu+3 more|arXiv (Cornell University)|Oct 5, 2022

Advanced Vision and Imaging4 citations

TL;DR

TartanCalib proposes an iterative wide-angle lens calibration pipeline that improves feature detection and subpixel refinement in highly distorted regions using adaptive subpixel refinement and intermediate camera model reprojection. It reduces reprojection error by up to 26.59% and detects 42.01% more features than state-of-the-art tools like Kalibr, significantly enhancing downstream tasks such as dense depth mapping.

ABSTRACT

Wide-angle cameras are uniquely positioned for mobile robots, by virtue of the rich information they provide in a small, light, and cost-effective form factor. An accurate calibration of the intrinsics and extrinsics is a critical pre-requisite for using the edge of a wide-angle lens for depth perception and odometry. Calibrating wide-angle lenses with current state-of-the-art techniques yields poor results due to extreme distortion at the edge, as most algorithms assume a lens with low to medium distortion closer to a pinhole projection. In this work we present our methodology for accurate wide-angle calibration. Our pipeline generates an intermediate model, and leverages it to iteratively improve feature detection and eventually the camera parameters. We test three key methods to utilize intermediate camera models: (1) undistorting the image into virtual pinhole cameras, (2) reprojecting the target into the image frame, and (3) adaptive subpixel refinement. Combining adaptive subpixel refinement and feature reprojection significantly improves reprojection errors by up to 26.59 %, helps us detect up to 42.01 % more features, and improves performance in the downstream task of dense depth mapping. Finally, TartanCalib is open-source and implemented into an easy-to-use calibration toolbox. We also provide a translation layer with other state-of-the-art works, which allows for regressing generic models with thousands of parameters or using a more robust solver. To this end, TartanCalib is the tool of choice for wide-angle calibration. Project website and code: http://tartancalib.com.

Motivation & Objective

• To address the poor feature detection and inaccurate calibration in highly distorted regions of wide-angle lenses, especially near image borders.
• To overcome the limitations of existing calibration pipelines that assume low-to-moderate distortion, violating the assumptions for extreme fisheye lenses.
• To develop a robust, iterative calibration pipeline that leverages intermediate camera models to improve feature detection and refinement accuracy.
• To enhance downstream robotic tasks such as visual odometry and dense depth mapping by enabling reliable use of the full image region, including highly distorted edges.
• To provide an open-source, easy-to-use calibration toolbox that integrates with existing tools like Kalibr and supports generic camera models.

Proposed method

• The pipeline uses an iterative process that alternates between undistorting images into virtual pinhole cameras and reprojecting the calibration target into the image frame using an intermediate camera model.
• It applies adaptive subpixel refinement based on the reprojected target size, which dynamically adjusts refinement sensitivity to distortion levels.
• A novel adaptive cornerSubPix() method improves feature localization accuracy in high-distortion regions, outperforming symmetry-based refinement, which is unstable under extreme distortion.
• The method combines intermediate model reprojection with subpixel refinement to iteratively improve camera parameter estimation and feature detection.
• The system supports integration with existing toolboxes like Kalibr via a translation layer, enabling regression of generic camera models with thousands of parameters.
• The pipeline is designed to be robust, requiring minimal hyperparameter tuning and running in 2–3 times the time of non-iterative baselines.

Experimental results

Research questions

• RQ1 Can iterative calibration with intermediate model refinement improve feature detection accuracy in highly distorted wide-angle lens regions?
• RQ2 Does adaptive subpixel refinement based on reprojected target size outperform traditional subpixel methods in extreme distortion scenarios?
• RQ3 To what extent does improved feature detection reduce reprojection error in wide-angle lens calibration?
• RQ4 How does enhanced feature coverage and accuracy affect downstream tasks such as stereo depth mapping?
• RQ5 Can the proposed pipeline be effectively integrated into existing calibration toolboxes like Kalibr without requiring major modifications?

Key findings

• TartanCalib reduces overall reprojection error by up to 26.59% compared to Kalibr on the Lensagon BF5M fisheye lens, with a drop from 0.3297 px to 0.2630 px.
• The method detects up to 42.01% more features than Kalibr, particularly in high-distortion border regions, significantly improving feature coverage.
• Adaptive subpixel refinement using cornerSubPix() outperforms symmetry-based refinement, which is unstable under high distortion, especially on the Lensagon BF5M dataset.
• The improved feature detection leads to less noisy disparity maps when generating virtual pinhole stereo pairs, demonstrating enhanced downstream performance in depth mapping.
• TartanCalib achieves state-of-the-art feature detection and calibration accuracy for wide-angle lenses, with minimal hyperparameter tuning and only 2–3× runtime overhead.
• The open-source toolbox is compatible with Kalibr and supports both generic camera models and robust solvers, enabling broad integration and use.