[Paper Review] Semantic Image Inpainting with Perceptual and Contextual Losses.

Raymond A. Yeh, Chen Chen+3 more|arXiv (Cornell University)|Jul 26, 2016

Generative Adversarial Networks and Image Synthesis38 references318 citations

TL;DR

This paper proposes a DCGAN-based image inpainting method using a combined loss function of contextual and perceptual losses to recover missing image regions with semantic accuracy and photorealistic quality. By optimizing this loss via back-propagation to map corrupted images to a latent space, the model generates realistic, semantically coherent outputs, outperforming existing methods on CelebA and SVHN under 80% random and blocky corruption.

ABSTRACT

In this paper, we propose a novel method for image inpainting based on a Deep Convolutional Generative Adversarial Network (DCGAN). We define a loss function consisting of two parts: (1) a contextual loss that preserves similarity between the input corrupted image and the recovered image, and (2) a perceptual loss that ensures a perceptually realistic output image. Given a corrupted image with missing values, we use back-propagation on this loss to map the corrupted image to a smaller latent space. The mapped vector is then passed through the generative model to predict the missing content. The proposed framework is evaluated on the CelebA and SVHN datasets for two challenging inpainting tasks with random 80% corruption and large blocky corruption. Experiments show that our method can successfully predict semantic information in the missing region and achieve pixel-level photorealism, which is impossible by almost all existing methods.

Motivation & Objective

• To address the challenge of recovering semantically meaningful and photorealistic content in missing image regions.
• To improve upon existing inpainting methods that fail to preserve both semantic structure and visual realism.
• To develop a loss function that balances structural similarity and perceptual quality in image reconstruction.
• To enable effective generation of missing content even under extreme corruption, such as 80% missing pixels or large blocky occlusions.

Proposed method

• The method employs a Deep Convolutional Generative Adversarial Network (DCGAN) for image generation.
• A contextual loss is defined to preserve similarity between the corrupted input and the recovered output in feature space.
• A perceptual loss is used to ensure the generated image matches real images in perceptual quality, using features from a pre-trained network.
• The model optimizes the combined loss via back-propagation to map the corrupted image to a latent vector.
• This latent vector is then decoded by the generator to produce the completed image.
• The framework is trained and evaluated on CelebA and SVHN datasets under two corruption scenarios: random 80% corruption and large blocky corruption.

Experimental results

Research questions

• RQ1 Can a combined loss of contextual and perceptual components improve semantic and perceptual quality in image inpainting?
• RQ2 How well does the method generalize to high levels of corruption, such as 80% missing pixels?
• RQ3 Can the model recover semantically meaningful content in missing regions where previous methods fail?
• RQ4 Does the use of back-propagation through the loss function enable effective latent space mapping for image completion?
• RQ5 How does the proposed method compare to existing approaches in terms of visual realism and structural fidelity?

Key findings

• The proposed method successfully recovers semantic content in missing regions, achieving results unattainable by almost all existing methods.
• The model produces pixel-level photorealism, significantly improving visual quality over prior approaches.
• Experiments on CelebA and SVHN demonstrate robust performance under both random 80% corruption and large blocky corruption.
• The combination of contextual and perceptual losses leads to better preservation of structural and perceptual consistency.
• The framework achieves state-of-the-art results in generating semantically coherent and visually realistic image completions.
• The method outperforms existing techniques in generating realistic textures and object-level structures in missing regions.