[Paper Review] Physics Informed and Data Driven Simulation of Underwater Images via Residual Learning

Tanmoy Mondal, Ricardo Mendoza+1 more|arXiv (Cornell University)|Feb 7, 2024

Seismic Imaging and Inversion TechniquesEarth and Planetary Sciences3 citations

TL;DR

This paper proposes a physics-informed, data-driven deep learning framework to simulate realistic underwater image degradation by combining a known image formation model with a residual learning network that captures unmodeled physical effects. The method estimates depth from RGB images, uses a complex image formation model for ground truth, and outperforms purely data-driven models in simulating realistic underwater distortions with improved interpretability and differentiability.

ABSTRACT

In general, underwater images suffer from color distortion and low contrast, because light is attenuated and backscattered as it propagates through water (differently depending on wavelength and on the properties of the water body). An existing simple degradation model (similar to atmospheric image "hazing" effects), though helpful, is not sufficient to properly represent the underwater image degradation because there are unaccounted for and non-measurable factors e.g. scattering of light due to turbidity of water, reflective characteristics of turbid medium etc. We propose a deep learning-based architecture to automatically simulate the underwater effects where only a dehazing-like image formation equation is known to the network, and the additional degradation due to the other unknown factors if inferred in a data-driven way. We only use RGB images (because in real-time scenario depth image is not available) to estimate the depth image. For testing, we have proposed (due to the lack of real underwater image datasets) a complex image formation model/equation to manually generate images that resemble real underwater images (used as ground truth). However, only the classical image formation equation (the one used for image dehazing) is informed to the network. This mimics the fact that in a real scenario, the physics are never completely known and only simplified models are known. Thanks to the ground truth, generated by a complex image formation equation, we could successfully perform a qualitative and quantitative evaluation of proposed technique, compared to other purely data driven approaches

Motivation & Objective

• To develop a realistic simulation of underwater image degradation that accounts for complex, unmeasurable physical factors beyond standard atmospheric models.
• To create a differentiable, interpretable deep learning emulator of underwater image formation that can be used for inverse problems like image restoration.
• To train a model using only RGB images and estimated depth, avoiding reliance on expensive RGB-D sensors in real-world scenarios.
• To generate a synthetic dataset of clean-degraded image pairs using a complex image formation model to serve as ground truth for training and evaluation.
• To outperform purely data-driven image-to-image translation models (e.g., Pix2Pix, CycleGAN) in simulating underwater image effects while maintaining physical plausibility.

Proposed method

• A deep neural network architecture combines a hard-coded classical image formation equation (similar to dehazing) with a residual learning branch to model unaccounted-for degradation factors.
• Depth maps are estimated from RGB images using a learned encoder-decoder network, enabling the physics-based component to be applied without direct depth input.
• A complex, multi-component image formation model is manually designed and used to generate realistic underwater image pairs as ground truth, simulating real-world degradation.
• The network is trained using a combination of adversarial, perceptual, and reconstruction losses to ensure realistic color, contrast, and structural fidelity.
• The residual learning component learns the difference between the simple physical model output and the ground truth, capturing missing scattering and turbidity effects.
• The trained model is differentiable via PyTorch, enabling its use in inverse problems such as underwater image restoration through optimization.

Experimental results

Research questions

• RQ1 Can a deep learning model effectively simulate underwater image degradation when only a simplified physical model and RGB images are available?
• RQ2 How well can a residual learning network capture unmeasurable physical effects like turbidity and scattering when trained on synthetic data generated by a complex image formation model?
• RQ3 To what extent does incorporating a known physical model improve the realism and interpretability of simulated underwater images compared to purely data-driven approaches?
• RQ4 Can the trained model serve as a differentiable emulator for solving inverse problems such as underwater image restoration?
• RQ5 How does the performance of the proposed method compare to state-of-the-art image-to-image translation models like Pix2Pix and CycleGAN in simulating underwater degradation?

Key findings

• The proposed method outperforms purely data-driven models like Pix2Pix and CycleGAN in simulating realistic underwater image degradation, particularly in capturing complex scattering and color distortion.
• Variant-2 of the proposed architecture consistently matches or exceeds the performance of the core technique and variant-1 across all metrics, indicating robustness to architectural variations.
• The model successfully simulates strong underwater blur and color shift effects, even when perfect color reconstruction is not achieved, demonstrating effective modeling of degradation.
• The use of a complex image formation model for generating ground truth enables high-quality training and reliable evaluation despite the lack of real underwater datasets.
• The trained model is fully differentiable, enabling its use as a differentiable physical emulator for inverse problems such as image restoration, a key advantage over black-box models.
• The method demonstrates that combining physics-informed priors with data-driven residual learning leads to more interpretable and generalizable simulators than purely data-driven alternatives.