[Paper Review] Learn from Unpaired Data for Image Restoration: A Variational Bayes Approach

Dihan Zheng, Xiaowen Zhang+2 more|arXiv (Cornell University)|Apr 21, 2022

Image and Signal Denoising Methods4 citations

TL;DR

This paper proposes LUD-VAE, a variational Bayes method that learns joint image restoration distributions from unpaired clean and corrupted images by modeling clean and noisy data as conditionally independent via shared latent representations. The approach enables training of effective image restoration models using synthetic paired data generated from unpaired data, achieving state-of-the-art performance in denoising, super-resolution, and low-light enhancement without requiring paired supervision.

ABSTRACT

Collecting paired training data is difficult in practice, but the unpaired samples broadly exist. Current approaches aim at generating synthesized training data from unpaired samples by exploring the relationship between the corrupted and clean data. This work proposes LUD-VAE, a deep generative method to learn the joint probability density function from data sampled from marginal distributions. Our approach is based on a carefully designed probabilistic graphical model in which the clean and corrupted data domains are conditionally independent. Using variational inference, we maximize the evidence lower bound (ELBO) to estimate the joint probability density function. Furthermore, we show that the ELBO is computable without paired samples under the inference invariant assumption. This property provides the mathematical rationale of our approach in the unpaired setting. Finally, we apply our method to real-world image denoising, super-resolution, and low-light image enhancement tasks and train the models using the synthetic data generated by the LUD-VAE. Experimental results validate the advantages of our method over other approaches.

Motivation & Objective

• To address the challenge of limited paired training data in real-world image restoration, where collecting paired clean-corrupted images is expensive and impractical.
• To develop a deep generative model that learns the joint distribution of clean and corrupted images from unpaired marginal data.
• To establish a theoretically grounded method for unpaired image restoration that avoids heuristic constraints like cycle consistency.
• To enable effective training of image restoration models using synthetic paired data generated from unpaired data via a probabilistic graphical model with inference invariance.
• To demonstrate that the evidence lower bound (ELBO) can be computed without paired samples under the inference invariant assumption, enabling end-to-end training on unpaired data.

Proposed method

• The method uses a probabilistic graphical model where clean images $\mathbf{x}$ and corrupted images $\mathbf{y}$ are conditionally independent given shared latent variables $\mathbf{z}$, with $\mathbf{z}$ encoding image content and $\mathbf{z}_n$ encoding degradation noise.
• A variational auto-encoder (VAE) framework is employed with separate encoders for $\mathbf{x}$ and $\mathbf{y}$, and a shared decoder that reconstructs $\mathbf{x}$ from $\mathbf{z}$, ensuring consistent latent representations across paired data.
• The inference invariant condition is imposed on $\mathbf{z}$, requiring that the posterior $q(\mathbf{z}|\mathbf{x})$ and $q(\mathbf{z}|\mathbf{y})$ remain approximately equal for paired $\mathbf{x}$ and $\mathbf{y}$, enabling ELBO computation from unpaired data.
• The ELBO is derived and shown to be computable without paired samples under the inference invariant assumption, providing a mathematically rigorous foundation for training on unpaired data.
• The model generates synthetic paired data by sampling from the learned joint distribution $p(\mathbf{x}, \mathbf{y})$, which is then used to train standard supervised image restoration networks.
• The method assumes that noise $\mathbf{n}_x$ and $\mathbf{n}_y$ are independent of input $\mathbf{x}$ and $\mathbf{y}$, respectively, and derives that $q(\mathbf{z}|\mathbf{x}) = \mathbb{E}_{p(\mathbf{n}_x)}[q(\mathbf{z}|\mathbf{x} + \mathbf{n}_x)]$, enabling robust inference under noise perturbations.

Experimental results

Research questions

• RQ1 Can a deep generative model learn the joint distribution of clean and corrupted images from unpaired data using variational inference?
• RQ2 Does the inference invariant condition on the shared latent space allow for ELBO optimization without paired data?
• RQ3 How well can synthetic paired data generated by the model support training of downstream image restoration networks?
• RQ4 What is the theoretical justification for computing the ELBO from unpaired data under the inference invariant assumption?
• RQ5 How does the method compare to GAN-based approaches in modeling degradation noise distributions?

Key findings

• The proposed LUD-VAE method enables effective training of image restoration models using only unpaired clean and corrupted images, eliminating the need for paired data collection.
• The inference invariant condition ensures that the posterior distributions $q(\mathbf{z}|\mathbf{x})$ and $q(\mathbf{z}|\mathbf{y})$ remain approximately equal for paired data, allowing the ELBO to be computed from unpaired samples.
• Theoretical analysis shows that $\|q(\mathbf{z}|\mathbf{x}) - q(\mathbf{z}|\mathbf{y})\|_1$ approaches zero as noise variances $\sigma_{\mathbf{x}}, \sigma_{\mathbf{y}} \to \infty$ with $\sigma_{\mathbf{x}}/\sigma_{\mathbf{y}} \to 1$, validating the inference invariance assumption in the limit.
• The method generates synthetic noise distributions that better match real noise than prior GAN-based methods like DeFlow, as shown in qualitative comparisons on AIM2019 dataset patches.
• Empirical results demonstrate state-of-the-art performance in image denoising, super-resolution, and low-light image enhancement, with significant improvements over existing unpaired learning baselines.
• The model achieves competitive results even when the degradation noise is complex and non-Gaussian, indicating robustness to real-world noise characteristics.