[Paper Review] Learning-Based Framework for Camera Calibration with Distortion Correction and High Precision Feature Detection

Yesheng Zhang, Xu Zhao+1 more|arXiv (Cornell University)|Feb 1, 2022

Optical measurement and interference techniques7 citations

TL;DR

This paper proposes a learning-based camera calibration framework that integrates deep learning for radial distortion correction, sub-pixel chessboard corner detection via heatmap learning with outlier rejection and collineation post-processing, and image-level RANSAC for stable parameter estimation. The method achieves higher precision and robustness than state-of-the-art approaches on both synthetic and real datasets.

ABSTRACT

Camera calibration is a crucial technique which significantly influences the performance of many robotic systems. Robustness and high precision have always been the pursuit of diverse calibration methods. State-of-the-art calibration techniques based on classical Zhang's method, however, still suffer from environmental noise, radial lens distortion and sub-optimal parameter estimation. Therefore, in this paper, we propose a hybrid camera calibration framework which combines learning-based approaches with traditional methods to handle these bottlenecks. In particular, this framework leverages learning-based approaches to perform efficient distortion correction and robust chessboard corner coordinate encoding. For sub-pixel accuracy of corner detection, a specially-designed coordinate decoding algorithm with embed outlier rejection mechanism is proposed. To avoid sub-optimal estimation results, we improve the traditional parameter estimation by RANSAC algorithm and achieve stable results. Compared with two widely-used camera calibration toolboxes, experiment results on both real and synthetic datasets manifest the better robustness and higher precision of the proposed framework. The massive synthetic dataset is the basis of our framework's decent performance and will be publicly available along with the code at https://github.com/Easonyesheng/CCS.

Motivation & Objective

• Address the persistent challenges of inexact corner detection, radial lens distortion, and unstable parameter estimation in camera calibration.
• Improve sub-pixel corner localization accuracy by combining CNN-based heatmap prediction with Gaussian surface fitting and outlier rejection.
• Correct radial distortion using a CNN trained on a large synthetic dataset to infer correction parameters before corner detection.
• Enhance parameter estimation stability by introducing an image-level RANSAC procedure based on reprojection error, reducing sensitivity to outliers.
• Develop a unified, end-to-end framework that integrates distortion correction, precise feature detection, and robust calibration for high-precision robotic and medical applications.

Proposed method

• Train a CNN to predict radial distortion correction parameters directly from distorted chessboard images using a practical distortion model.
• Design a heatmap representation where each corner is modeled as a Gaussian distribution centered at the ground-truth sub-pixel location for supervision.
• Use a deep CNN to regress the heatmap from input images, then apply Gaussian surface fitting to refine corner locations to sub-pixel accuracy.
• Apply distribution-aware outlier rejection to filter unreliable detections based on abnormal heatmap distributions.
• Perform collineation post-processing to enforce geometric consistency among detected corners by leveraging line collinearity in the chessboard pattern.
• Integrate an image-level RANSAC algorithm that evaluates reprojection error across all detected corners to select the optimal camera model, improving parameter estimation stability.

Experimental results

Research questions

• RQ1 Can a deep learning-based approach improve sub-pixel corner detection accuracy beyond pixel-level precision in the presence of noise and distortion?
• RQ2 To what extent can a CNN effectively learn and correct radial lens distortion without explicit geometric priors?
• RQ3 How does integrating outlier rejection and geometric post-processing enhance the robustness of corner detection in challenging imaging conditions?
• RQ4 Can a simplified image-level RANSAC procedure significantly improve the stability and accuracy of camera parameter estimation compared to purely algebraic optimization?
• RQ5 Does a unified framework combining distortion correction, precise feature detection, and robust estimation outperform existing state-of-the-art calibration methods?

Key findings

• The proposed framework achieves superior sub-pixel corner detection accuracy by combining heatmap learning with Gaussian fitting and outlier rejection, outperforming prior CNN-based methods.
• The CNN-based distortion correction effectively removes radial distortion, enabling more accurate corner detection in highly distorted images.
• The distribution-aware outlier rejection step successfully identifies and removes unreliable detections with abnormal heatmap shapes, improving detection reliability.
• The collineation post-processing step refines corner coordinates by enforcing geometric consistency, further enhancing precision.
• The image-level RANSAC procedure significantly improves the stability of camera parameter estimation by minimizing reprojection error across all corners.
• On both synthetic and real datasets, the framework demonstrates higher precision and greater robustness than state-of-the-art methods, with quantitative results showing improved mean and standard deviation of re-projection error.