[Paper Review] Optimizing Generative Adversarial Networks for Image Super Resolution via Latent Space Regularization

Sheng Zhong, Shifu Zhou|arXiv (Cornell University)|Jan 22, 2020

Advanced Image Processing Techniques27 references4 citations

TL;DR

This paper proposes Latent Space Regularization (LSR) for Generative Adversarial Networks (GANs) in single image super-resolution (SISR), enforcing Lipschitz continuity via a learned encoding network and latent space regularization terms to improve structural fidelity and reduce artifacts. The method outperforms state-of-the-art models like ESRGAN and SRGAN in perceptual quality, sharpness, and structural similarity, with lower L1 error and improved consistency across benchmarks.

ABSTRACT

Natural images can be regarded as residing in a manifold that is embedded in a higher dimensional Euclidean space. Generative Adversarial Networks (GANs) try to learn the distribution of the real images in the manifold to generate samples that look real. But the results of existing methods still exhibit many unpleasant artifacts and distortions even for the cases where the desired ground truth target images are available for supervised learning such as in single image super resolution (SISR). We probe for ways to alleviate these problems for supervised GANs in this paper. We explicitly apply the Lipschitz Continuity Condition (LCC) to regularize the GAN. An encoding network that maps the image space to a new optimal latent space is derived from the LCC, and it is used to augment the GAN as a coupling component. The LCC is also converted to new regularization terms in the generator loss function to enforce local invariance. The GAN is optimized together with the encoding network in an attempt to make the generator converge to a more ideal and disentangled mapping that can generate samples more faithful to the target images. When the proposed models are applied to the single image super resolution problem, the results outperform the state of the art.

Motivation & Objective

• To address persistent artifacts and structural distortions in GAN-based image super-resolution despite access to ground truth targets.
• To improve the faithfulness of generated high-resolution images to the ground truth by enforcing local invariance in the latent space.
• To enhance training stability and convergence of GANs in supervised SISR by incorporating Lipschitz continuity constraints.
• To develop a joint optimization framework between the generator and an encoding network that maps images to a disentangled, optimal latent space.
• To outperform existing SISR methods like ESRGAN and SRGAN in perceptual quality and structural accuracy.

Proposed method

• The method introduces a coupling encoding network that maps high-resolution images to a lower-dimensional latent space, derived from the Lipschitz Continuity Condition (LCC).
• The LCC is reformulated into regularization terms in the generator loss function using the Karush-Kuhn-Tucker (KKT) condition to enforce local invariance.
• The generator and encoding network are jointly optimized to ensure the generator learns a more ideal, disentangled mapping to the target image manifold.
• The approach integrates the LCC-based regularization into the GAN framework, particularly enhancing the ESRGAN architecture with Relativistic adversarial loss and perceptual loss.
• The method uses VGG19-conv54 features for perceptual loss and applies L1 normalization to the Y channel for quantitative evaluation.
• The model is trained end-to-end on the DIV2K dataset, with ablation studies on different loss components and adversarial objectives.

Experimental results

Research questions

• RQ1 Can enforcing Lipschitz continuity in the latent space reduce structural distortions and artifacts in GAN-based super-resolution?
• RQ2 Does joint optimization of a learned encoding network with the GAN improve the faithfulness of generated images to ground truth?
• RQ3 How does LSR compare to standard GANs and ESRGAN in terms of perceptual quality and structural similarity?
• RQ4 Does the LSR method improve consistency and reduce variance in quantitative metrics like PSNR, SSIM, and PI?
• RQ5 Can the LSR framework be generalized to other image restoration tasks beyond SISR?

Key findings

• The LSR-based GANs achieve superior perceptual quality and structural fidelity compared to ESRGAN and SRGAN, with clearer textures and fewer artifacts in visual results.
• The CLSR model reduces the average L1 error to 0.420 on BSD100, 0.429 on PIRM, 0.436 on Urban, and 0.444 on Set14, outperforming CESR across all datasets.
• The PSNR, SSIM, and PI standard deviations are lower in LSR-based models, indicating more consistent performance across test sets.
• The CLSR model corrects structural distortions such as smeared zebra stripes and spurious lines in chimneys, which persist in ESRGAN and CESR results.
• The LSR method performs better with relativistic adversarial losses and cosine similarity-based contextual loss, indicating sensitivity to the choice of adversarial objective.
• Despite using only the DIV2K dataset, the LSR model achieves state-of-the-art performance, suggesting strong generalization and robustness.