[Paper Review] FINO: Flow-based Joint Image and Noise Model

Lanqing Guo, Siyu Huang+2 more|arXiv (Cornell University)|Nov 11, 2021

Image and Signal Denoising Methods29 references4 citations

TL;DR

FINO proposes a flow-based generative model that jointly learns image and noise distributions by explicitly disentangling clean images and noise in a latent space via invertible normalizing flows. By incorporating noise correlation constraints and structural alignment through variable swapping, FINO achieves state-of-the-art performance on both synthetic and real-world denoising benchmarks, with superior generalization to spatially variant and inaccurately estimated noise.

ABSTRACT

One of the fundamental challenges in image restoration is denoising, where the objective is to estimate the clean image from its noisy measurements. To tackle such an ill-posed inverse problem, the existing denoising approaches generally focus on exploiting effective natural image priors. The utilization and analysis of the noise model are often ignored, although the noise model can provide complementary information to the denoising algorithms. In this paper, we propose a novel Flow-based joint Image and NOise model (FINO) that distinctly decouples the image and noise in the latent space and losslessly reconstructs them via a series of invertible transformations. We further present a variable swapping strategy to align structural information in images and a noise correlation matrix to constrain the noise based on spatially minimized correlation information. Experimental results demonstrate FINO's capacity to remove both synthetic additive white Gaussian noise (AWGN) and real noise. Furthermore, the generalization of FINO to the removal of spatially variant noise and noise with inaccurate estimation surpasses that of the popular and state-of-the-art methods by large margins.

Motivation & Objective

• To address the limitation of existing denoising methods that focus solely on image priors while neglecting explicit noise modeling.
• To develop a unified framework that jointly models the distributions of clean images and noise, leveraging complementary information from both.
• To enable effective disentanglement of image and noise components in a latent space through invertible transformations.
• To improve generalization to real-world noise and non-uniform noise by incorporating noise correlation constraints and structural alignment.
• To achieve blind denoising without requiring external noise estimation, enhancing practical applicability.

Proposed method

• Employ a flow-based invertible neural network to perform lossless bijective transformations between the noisy image and a latent space where image and noise components are disentangled.
• Introduce a variable swapping strategy to align structural content between the reconstructed image and the ground truth, improving feature-level consistency.
• Apply a noise correlation matrix constraint that enforces spatially independent noise across neighboring regions, modeling noise as uncorrelated.
• Train the model using a multi-component loss: reconstruction loss for image and noise, content loss for structural alignment, and noise modeling loss for distributional consistency.
• Decouple the training process by jointly optimizing image prior learning and noise distribution modeling in a single end-to-end framework.
• Enable blind denoising by removing the need for explicit noise level estimation through robust joint modeling.

Experimental results

Research questions

• RQ1 Can explicit joint modeling of image and noise distributions improve denoising performance beyond relying solely on image priors?
• RQ2 How effectively can invertible flows disentangle image and noise components in a shared latent space while preserving structural fidelity?
• RQ3 To what extent does enforcing spatially uncorrelated noise modeling improve robustness to real-world and non-uniform noise?
• RQ4 Does the proposed variable swapping strategy enhance structural alignment and generalization across varying noise levels?
• RQ5 Can FINO generalize effectively to real-world noise without requiring noise level estimation or external noise modeling?

Key findings

• FINO achieves a PSNR of 39.40 and SSIM of 0.957 on the real-world SIDD dataset, outperforming state-of-the-art methods including InvDN (39.28/0.955) and AINDNet (39.08/0.955).
• On synthetic AWGN with σ=25, FINO achieves a PSNR of 31.82, demonstrating strong performance and generalization across noise levels (σ=20, 25, 30).
• The ablation study confirms that removing the content alignment loss or reconstruction loss significantly degrades performance, especially at higher noise levels.
• Visualizations show that the reconstructed noise closely matches the ground truth, confirming effective noise disentanglement and modeling.
• FINO generalizes well to spatially variant and inaccurately estimated noise, outperforming methods that rely on fixed noise assumptions.
• With only 3.96M parameters, FINO achieves superior performance with a much smaller model size than competing methods like AINDNet (13.76M) and VDN (7.81M).