[Paper Review] Generative Image Inpainting with Contextual Attention

Jiahui Yu, Zhe Lin+4 more|arXiv (Cornell University)|Jan 24, 2018

Generative Adversarial Networks and Image Synthesis37 references214 citations

TL;DR

Introduces a two-stage generative inpainting network with a novel contextual attention layer that explicitly borrows distant background patches to fill missing regions, achieving higher-quality results across faces, textures, and natural images.

ABSTRACT

Recent deep learning based approaches have shown promising results for the challenging task of inpainting large missing regions in an image. These methods can generate visually plausible image structures and textures, but often create distorted structures or blurry textures inconsistent with surrounding areas. This is mainly due to ineffectiveness of convolutional neural networks in explicitly borrowing or copying information from distant spatial locations. On the other hand, traditional texture and patch synthesis approaches are particularly suitable when it needs to borrow textures from the surrounding regions. Motivated by these observations, we propose a new deep generative model-based approach which can not only synthesize novel image structures but also explicitly utilize surrounding image features as references during network training to make better predictions. The model is a feed-forward, fully convolutional neural network which can process images with multiple holes at arbitrary locations and with variable sizes during the test time. Experiments on multiple datasets including faces (CelebA, CelebA-HQ), textures (DTD) and natural images (ImageNet, Places2) demonstrate that our proposed approach generates higher-quality inpainting results than existing ones. Code, demo and models are available at: https://github.com/JiahuiYu/generative_inpainting.

Motivation & Objective

• Motivate the need for better long-range contextual modeling in image inpainting.
• Propose a unified feed-forward network with a contextual attention module to borrow information from distant regions.
• Improve training stability and speed with enhanced loss functions and a two-stage coarse-to-fine architecture.
• Show applicability across diverse datasets including CelebA, CelebA-HQ, DTD, ImageNet, and Places2.

Proposed method

• Propose a two-stage coarse-to-fine network where the coarse stage reconstructs missing content and the refinement stage improves results.
• Introduce a novel contextual attention layer that matches foreground patches to background patches using cosine similarity, softmax weighting, and deconvolution to reconstruct patches.
• Use two Wasserstein GAN losses (global and local) with a reconstruction loss to stabilize training and enforce both global and local fidelity.
• Implement spatially discounted reconstruction loss to reduce over-penalization near hole centers and facilitate learning.
• Adopt memory-efficient strategies for the contextual attention mechanism, including patch sampling/strides and optional downscaling of inputs.
• Train end-to-end with a combination of reconstruction loss and dual GAN objectives, achieving faster convergence and better visual quality.

Experimental results

Research questions

• RQ1 Can a contextual attention mechanism enable explicit borrowing of distant background patches to improve inpainting quality?
• RQ2 Does a two-stage coarse-to-fine generative framework with global and local adversarial supervision outperform prior inpainting models?
• RQ3 How do spatially discounted reconstruction loss and attention-based fusion affect training stability and final image fidelity?
• RQ4 Is the proposed method effective across diverse data domains such as faces, textures, and natural scenes?

Key findings

• The full model with contextual attention yields more realistic inpainting with fewer artifacts than the baseline model across multiple datasets.
• Attention maps visualize which background patches are most relevant for filling each foreground pixel, indicating successful contextual borrowing.
• Quantitative results on Places2 show: PatchMatch 16.1% ell1, 3.9% ell2, PSNR 16.62, TV 25.0%; Baseline 9.4% ell1, 2.4% ell2, PSNR 18.15, TV 25.7%; Our method 8.6% ell1, 2.1% ell2, PSNR 18.91, TV 25.3%.
• The proposed two-stage network and contextual attention enable faster training and reduce the need for post-processing like image blending.
• The model achieves good generalization across CelebA, CelebA-HQ, DTD, ImageNet, and Places2 datasets.