[Paper Review] Stereo camera system calibration: the need of two sets of parameters

Riccardo Beschi, Xiao Feng+3 more|arXiv (Cornell University)|Jan 14, 2021

Advanced Vision and Imaging18 references4 citations

TL;DR

This paper proposes calibrating a stereo camera system twice—once to minimize 2D reprojection error and once to minimize 3D reconstruction error—demonstrating that a single set of parameters cannot optimally serve both tasks due to image noise. The key finding is that separate parameter sets yield superior performance in correspondence matching and 3D reconstruction, respectively, resolving a fundamental trade-off in stereo vision calibration.

ABSTRACT

The reconstruction of a scene via a stereo-camera system is a two-steps process, where at first images from different cameras are matched to identify the set of point-to-point correspondences that then will actually be reconstructed in the three dimensional real world. The performance of the system strongly relies of the calibration procedure, which has to be carefully designed to guarantee optimal results. We implemented three different calibration methods and we compared their performance over 19 datasets. We present the experimental evidence that, due to the image noise, a single set of parameters is not sufficient to achieve high accuracy in the identification of the correspondences and in the 3D reconstruction at the same time. We propose to calibrate the system twice to estimate two different sets of parameters: the one obtained by minimizing the reprojection error that will be used when dealing with quantities defined in the 2D space of the cameras, and the one obtained by minimizing the reconstruction error that will be used when dealing with quantities defined in the real 3D world.

Motivation & Objective

• To investigate whether a single calibration set can optimally support both 2D correspondence matching and 3D reconstruction in stereo vision systems.
• To identify the limitations of existing calibration methods that use one parameter set for both tasks under real-world image noise.
• To evaluate the performance trade-offs between minimizing reprojection error (2D) and reconstruction error (3D) in stereo calibration.
• To propose and validate a dual-calibration approach that separates the optimization objectives for improved overall system accuracy.

Proposed method

• Implementing three calibration methods: essential matrix estimation, 2D minimization (minimizing reprojection error), and 3D minimization (minimizing reconstruction error).
• Using the essential matrix method as initialization for both 2D and 3D minimization via a Monte Carlo optimization strategy.
• Calibrating the system twice: once to minimize 2D reprojection error for correspondence identification, and once to minimize 3D reconstruction error for 3D output.
• Evaluating all three methods on 19 datasets to compare performance across correspondence matching and 3D reconstruction accuracy.
• Applying the 2D minimization parameters for 2D tasks (e.g., correspondence matching) and the 3D minimization parameters for 3D reconstruction tasks.
• Using the pinhole camera model and projective geometry to define the relationship between 3D world points and their 2D image projections.

Experimental results

Research questions

• RQ1 Can a single set of calibration parameters achieve high accuracy in both 2D correspondence matching and 3D reconstruction under real-world image noise?
• RQ2 What is the performance trade-off between minimizing 2D reprojection error and minimizing 3D reconstruction error in stereo camera calibration?
• RQ3 Does separating the calibration process into two distinct optimizations—targeting 2D and 3D accuracy respectively—lead to better overall system performance?
• RQ4 How do different calibration strategies compare in terms of reprojection error and reconstruction error across diverse datasets?

Key findings

• The 2D minimization method achieved the best performance in identifying correct point-to-point correspondences, as measured by low reprojection error.
• The 3D minimization method produced the most accurate 3D reconstructions, with reconstruction errors consistently below 0.01 across all 19 datasets.
• Using a single set of parameters resulted in suboptimal performance for both correspondence matching and 3D reconstruction due to conflicting noise assumptions.
• The 2D minimization method assumes all noise is in 3D space, while the 3D minimization method assumes all noise is in 2D image space, leading to inherent trade-offs.
• The dual-calibration approach—using separate parameters for 2D and 3D tasks—achieves superior results than any single-parameter method.
• The study confirms that minimizing reprojection error and minimizing reconstruction error are conflicting objectives, necessitating two distinct calibration procedures.