[Paper Review] Unconstrained Two-parallel-plane Model for Focused Plenoptic Cameras Calibration

Chunping Zhang, Zhe Ji+1 more|arXiv (Cornell University)|Aug 16, 2016

Advanced Vision and Imaging17 references3 citations

TL;DR

This paper proposes a novel 7-parameter unconstrained two-parallel-plane (TPP) model to calibrate focused plenoptic cameras by establishing a projective transformation between 3D scene points and 2D image sensor measurements. The method uses a closed-form solution followed by nonlinear optimization via Levenberg-Marquardt to minimize re-projection error, achieving sub-pixel accuracy (RMS error < 0.3 pixels) on both simulated and real data.

ABSTRACT

The plenoptic camera can capture both angular and spatial information of the rays, enabling 3D reconstruction by single exposure. The geometry of the recovered scene structure is affected by the calibration of the plenoptic camera significantly. In this paper, we propose a novel unconstrained two-parallel-plane (TPP) model with 7 parameters to describe a 4D light field. By reconstructing scene points from ray-ray association, a 3D projective transformation is deduced to establish the relationship between the scene structure and the TPP parameters. Based on the transformation, we simplify the focused plenoptic camera as a TPP model and calibrate its intrinsic parameters. Our calibration method includes a close-form solution and a nonlinear optimization by minimizing re-projection error. Experiments on both simulated data and real scene data verify the performance of the calibration on the focused plenoptic camera.

Motivation & Objective

• Address the limitations of existing plenoptic camera calibration models, which suffer from redundant or incomplete parameters and poor initialization.
• Develop a concise, physically meaningful model that accurately describes the 4D light field geometry in focused plenoptic cameras.
• Establish a robust calibration pipeline that combines linear initialization with nonlinear optimization to improve parameter accuracy.
• Enable high-precision 3D reconstruction and digital refocusing by accurately decoding metric 4D light field data from raw sensor images.

Proposed method

• Propose a 7-parameter unconstrained TPP model to represent the 4D light field, where two parallel planes define the optical path and ray geometry.
• Derive a 3D projective transformation matrix that maps scene points to image sensor pixels using the TPP parameters, forming the theoretical basis for calibration.
• Implement a closed-form solution for initial parameter estimation using linear relations derived from ray-ray associations and 3D point reconstruction.
• Apply nonlinear optimization via the Levenberg-Marquardt algorithm to refine parameters by minimizing re-projection error on the raw image plane.
• Use a calibration board with multiple poses to collect data for robust parameter estimation, with preprocessing to extract micro-lens sub-aperture images.
• Validate the calibration by rendering refocused images using ray tracing, demonstrating the method’s utility in downstream applications.

Experimental results

Research questions

• RQ1 How can a simplified, physically consistent 4D light field model be constructed for focused plenoptic cameras to improve calibration accuracy?
• RQ2 Can a closed-form solution for intrinsic parameters be derived from a projective transformation between scene points and image measurements?
• RQ3 To what extent does the proposed method reduce re-projection error compared to existing calibration techniques?
• RQ4 How does the number of calibration poses and the number of calibration points affect the robustness and accuracy of the calibration?
• RQ5 Can the calibrated model support high-quality digital refocusing and 3D reconstruction in real-world scenarios?

Key findings

• The proposed 7-parameter unconstrained TPP model effectively captures the geometric relationship between 3D scene points and 2D image sensor measurements in focused plenoptic cameras.
• The calibration method achieves a re-projection error of less than 0.3 pixels (RMS) when using at least 5 calibration poses, with consistent performance across different point configurations.
• The RMS error decreases from 0.31505 to 0.27846 pixels when distortion coefficients are included in the optimization, demonstrating the effectiveness of nonlinear refinement.
• With 9 poses and 8×10 calibration points, the method achieves an RMS error of 0.27846 pixels, confirming high precision and robustness.
• The estimated poses and ray directions enable accurate digital refocusing, as validated by rendered images showing consistent and sharp focus across different views.
• The method outperforms prior approaches by avoiding assumptions about micro-lens center alignment and providing a more accurate, physically grounded calibration framework.