[Paper Review] Image Inpainting via Generative Multi-column Convolutional Neural Networks

Yi Wang, Xin Tao+3 more|arXiv (Cornell University)|Oct 20, 2018

Generative Adversarial Networks and Image Synthesis25 references87 citations

TL;DR

The paper introduces a Generative Multi-column Convolutional Neural Network (GMCNN) for image inpainting, using multiple parallel branches, ID-MRF regularization, and a confidence-driven reconstruction loss to synthesize plausible global structures and local textures without post-processing.

ABSTRACT

In this paper, we propose a generative multi-column network for image inpainting. This network synthesizes different image components in a parallel manner within one stage. To better characterize global structures, we design a confidence-driven reconstruction loss while an implicit diversified MRF regularization is adopted to enhance local details. The multi-column network combined with the reconstruction and MRF loss propagates local and global information derived from context to the target inpainting regions. Extensive experiments on challenging street view, face, natural objects and scenes manifest that our method produces visual compelling results even without previously common post-processing.

Motivation & Objective

• Motivate improved inpainting by capturing both global structure and local texture.
• Propose a multi-branch generator to extract diverse components from full-resolution inputs.
• Introduce ID-MRF regularization to diversify and constrain texture generation.
• Design a confidence-driven reconstruction loss to enforce spatially adaptive constraints.
• Demonstrate state-of-the-art results across street scenes, faces, and natural images without post-processing.

Proposed method

• Use a Generative Multi-column CNN with three parallel encoder–decoder branches that operate on full-resolution inputs.
• Concatenate branch outputs and pass through a shared decoder to produce the inpainted image.
• Introduce ID-MRF regularization computed over VGG feature layers to encourage diverse and realistic textures in training.
• Employ a confidence-driven reconstruction loss that weights pixels based on distance to missing regions and boundary proximity.
• Incorporate adversarial losses with global and local discriminators via Wasserstein GAN with gradient penalty during training.

Experimental results

Research questions

• RQ1 How can multiple parallel encoders with different receptive fields improve inpainting of global structure and local texture?
• RQ2 Can an implicit, diversified MRF regularization during training produce more realistic textures than post-processing MRF steps?
• RQ3 Does a spatially variant reconstruction loss enhance boundary consistency and texture quality across various masks and scenes?
• RQ4 What is the impact of combining ID-MRF, confidence-driven loss, and adversarial training on quantitative and perceptual inpainting quality?

Key findings

• The GMCNN with varied receptive fields outperforms single-branch and coarse-to-fine architectures in visual fidelity.
• ID-MRF regularization improves local texture details and texture diversity compared to not using it.
• Confidence-driven reconstruction loss yields better boundary handling and gradual learning focus than spatially discounted alternatives.
• Quantitative results show competitive PSNR and SSIM across multiple datasets, with user studies favoring GMCNN outputs over baselines.
• Visual results demonstrate strong performance on street scenes, faces, and natural objects without post-processing.