[Paper Review] Accurate smartphone camera simulation using 3D scenes

Zheng Lyu, Thomas Goossens+2 more|arXiv (Cornell University)|Jan 19, 2022

Image and Video Quality Assessment4 citations

TL;DR

This paper presents an end-to-end, physically based simulation of smartphone camera imaging using high-dynamic-range 3D scenes, accurately modeling optical blur, depth of field, spectral response, inter-reflections, and sensor noise. Validation against a real Cornell Box setup shows close agreement between simulated and measured images, confirming high accuracy for use in camera design and synthetic data generation for machine learning.

ABSTRACT

We assess the accuracy of a smartphone camera simulation. The simulation is an end-to-end analysis that begins with a physical description of a high dynamic range 3D scene and includes a specification of the optics and the image sensor. The simulation is compared to measurements of a physical version of the scene. The image system simulation accurately matched measurements of optical blur, depth of field, spectral quantum efficiency, scene inter-reflections, and sensor noise. The results support the use of image systems simulation methods for soft prototyping cameras and for producing synthetic data in machine learning applications.

Motivation & Objective

• To develop a high-fidelity, end-to-end simulation of smartphone camera imaging using physically based ray tracing.
• To validate the simulation accuracy against real measurements from a constructed physical Cornell Box.
• To enable reliable soft prototyping of imaging sensors and synthetic data generation for machine learning applications.
• To model complex scene effects such as spectral radiance, inter-reflections, and sensor noise with physical precision.

Proposed method

• The simulation uses physically based ray tracing to compute spectral irradiance in high-dynamic-range 3D scenes, including surface inter-reflections and shadows.
• It models the complete imaging pipeline: scene radiance, optics, sensor response, and digital output with physical parameters.
• The sensor model incorporates photon shot noise, PRNU (pixel response non-uniformity), and conversion gain from electrons to digital values.
• Key equations model the expected signal and variance, with conversion gain estimated from uniform scene measurements using Equation (12).
• A 10×10 pixel sampling strategy is used to identify uniform regions in images, rejecting non-uniform or noisy regions based on statistical criteria.
• The simulation is validated by comparing digital values and noise characteristics against real images captured from a physical Cornell Box.

Experimental results

Research questions

• RQ1 Can an end-to-end, physically based simulation accurately reproduce optical blur and depth of field in a real smartphone camera?
• RQ2 To what extent does the simulation match real sensor noise, including photon shot noise and PRNU?
• RQ3 How accurately does the simulation model spectral quantum efficiency and indirect lighting from surface inter-reflections?
• RQ4 Can the simulation replicate complex scene effects like shadows and color bleeding in a controlled 3D environment?
• RQ5 How well does the simulated sensor response match measured data from a physical camera under uniform illumination?

Key findings

• The simulation accurately matched measured optical blur and depth of field in the physical Cornell Box setup.
• Spectral quantum efficiency and inter-reflection effects in the simulation closely matched real-world measurements.
• The estimated conversion gain (0.1677 dv/e⁻) was within 1.8% of the manufacturer’s value (0.1707 dv/e⁻), confirming accurate sensor modeling.
• After applying the estimated conversion gain, the distribution of estimated electrons matched Poisson statistics, validating noise modeling.
• The variance of digital values in the simulated and measured images showed strong agreement, particularly after accounting for PRNU and gain effects.
• The method successfully identified uniform regions in both simulated and real images, with rejection criteria based on spatial non-uniformity and deviation from expected variance.