[Paper Review] Deep Learning for Camera Calibration and Beyond: A Survey

Kang Liao, Lang Nie+6 more|arXiv (Cornell University)|Mar 19, 2023

Optical measurement and interference techniques26 citations

TL;DR

This paper surveys learning-based camera calibration methods across standard, distortion, cross-view, and cross-sensor models, compiling datasets and benchmarks and providing an open-source repository.

ABSTRACT

Camera calibration involves estimating camera parameters to infer geometric features from captured sequences, which is crucial for computer vision and robotics. However, conventional calibration is laborious and requires dedicated collection. Recent efforts show that learning-based solutions have the potential to be used in place of the repeatability works of manual calibrations. Among these solutions, various learning strategies, networks, geometric priors, and datasets have been investigated. In this paper, we provide a comprehensive survey of learning-based camera calibration techniques, by analyzing their strengths and limitations. Our main calibration categories include the standard pinhole camera model, distortion camera model, cross-view model, and cross-sensor model, following the research trend and extended applications. As there is no unified benchmark in this community, we collect a holistic calibration dataset that can serve as a public platform to evaluate the generalization of existing methods. It comprises both synthetic and real-world data, with images and videos captured by different cameras in diverse scenes. Toward the end of this paper, we discuss the challenges and provide further research directions. To our knowledge, this is the first survey for the learning-based camera calibration (spanned 10 years). The summarized methods, datasets, and benchmarks are available and will be regularly updated at https://github.com/KangLiao929/Awesome-Deep-Camera-Calibration.

Motivation & Objective

• Provide a comprehensive overview of learning-based camera calibration methods and their evolution over eight years.
• Categorize methods by calibration objective (intrinsics, extrinsics, cross-view, cross-sensor) and by extended applications.
• Assess strengths, limitations, and practical challenges of learning-based calibration across diverse cameras and scenes.
• Compile a holistic dataset and benchmarks to evaluate generalization of existing methods.
• Offer open-source resources and guidance to foster ongoing research in this field.

Proposed method

• Describe and compare two main learning paradigms: regression-based calibration and reconstruction-based calibration.
• Review learning strategies (supervised, unsupervised, semi-/weak/self-supervised, reinforcement) and their applicability to calibration tasks.
• Summarize methods by calibration objective and camera model (standard, distortion, cross-view, cross-sensor) with architecture, loss, dataset, and metrics.
• Compile a public dataset collection comprising synthetic and real-world data for broad evaluation.
• Provide an open-source repository with a taxonomy of reviewed works and benchmarks for ongoing updates.

Experimental results

Research questions

• RQ1 What are the dominant learning-based paradigms for camera calibration and how do they differ in practice?
• RQ2 How have learning-based methods been applied to intrinsic, extrinsic, distortion, cross-view, and cross-sensor calibration tasks?
• RQ3 What datasets, benchmarks, and evaluation metrics exist or are needed to assess generalization of these methods?
• RQ4 What are the key challenges and future directions for learning-based camera calibration in diverse environments and camera models?

Key findings

• The survey covers over 100 papers on learning-based camera calibration from 2015 to 2022.
• Two primary learning paradigms are identified: regression-based and reconstruction-based calibration.
• A wide range of learning strategies (supervised, unsupervised, semi-/weak/self-supervised, reinforcement) are explored across methods.
• The authors collect a holistic calibration dataset (synthetic and real-world) to evaluate generalization of methods.
• An open-source repository with a taxonomy and benchmarks is provided and regularly updated.
• The paper discusses challenges and proposes future research directions in learning-based camera calibration.