[Paper Review] Patch-Based Image Inpainting with Generative Adversarial Networks

Uğur Demir, Gözde Ünal|arXiv (Cornell University)|Mar 20, 2018

Advanced Image Processing Techniques6 references94 citations

TL;DR

Demir and Unal propose PGGAN, a GAN-based inpainting model that combines global (G-GAN) and patch-based (PatchGAN) discriminators with shared early layers to capture both global structure and local texture for high-quality large-hole inpainting.

ABSTRACT

Area of image inpainting over relatively large missing regions recently advanced substantially through adaptation of dedicated deep neural networks. However, current network solutions still introduce undesired artifacts and noise to the repaired regions. We present an image inpainting method that is based on the celebrated generative adversarial network (GAN) framework. The proposed PGGAN method includes a discriminator network that combines a global GAN (G-GAN) architecture with a patchGAN approach. PGGAN first shares network layers between G-GAN and patchGAN, then splits paths to produce two adversarial losses that feed the generator network in order to capture both local continuity of image texture and pervasive global features in images. The proposed framework is evaluated extensively, and the results including comparison to recent state-of-the-art demonstrate that it achieves considerable improvements on both visual and quantitative evaluations.

Motivation & Objective

• Motivate improved inpainting quality for large missing regions by capturing both global image structure and local texture details.
• Propose a discriminator that combines global GAN and PatchGAN with shared and split pathways.
• Enhance generator training with dilated and interpolated convolutions to improve texture synthesis and reduce artifacts.
• Perform ablation studies to understand component contributions and compare against state-of-the-art methods.

Proposed method

• Use a ResNet-based generator with down-sampling, residual blocks, and up-sampling; employ dilated convolutions and interpolated convolutions to enlarge receptive fields and avoid checkerboard artifacts.
• Design PGGAN: a discriminator that shares early layers for G-GAN and PatchGAN paths, then splits into two outputs for whole-image realism and local patch realism.
• Train with a joint loss: reconstruction (L1), g_adv (G-GAN path), p_adv (PatchGAN path), balanced by weights in the total loss.
• Incorporate three loss components: L_rec = L1 distance, L_GAN, and L_pGAN, optimized with Adam.
• Evaluate on Paris Street View, Google Street View, and Places datasets with 256x256 and 512x512 image sizes.

Experimental results

Research questions

• RQ1 Can combining global image realism with patch-level texture realism improve inpainting quality for large missing regions?
• RQ2 Do shared early layers plus dual adversarial paths better enforce global coherence and local texture?
• RQ3 What is the impact of dilated/interpolated convolutions on texture synthesis and artifact reduction in inpainting?
• RQ4 How does PGGAN compare to state-of-the-art inpainting methods in terms of quantitative metrics and perceptual quality?

Key findings

• PGGAN achieves better PSNR and SSIM as well as lower L1/L2 losses than CE and GLGAN on 256x256 Paris Street View.
• On 256x256 Paris Street View, PGGAN-DRes and PGGAN-Res surpass CE and GLGAN in PSNR and SSIM, with lower reconstruction errors.
• For 512x512 Paris Street View, PGGAN-DRes outperforms NPS, achieving lower L1/L2 losses and higher PSNR and SSIM.
• Perceptual evaluations favor PGGAN variants over CE and GLGAN, suggesting improved naturalness of inpainted results.
• Ablation studies show combining G-GAN and PatchGAN discriminators yields visually and quantitatively superior results compared to using either alone.