[Paper Review] Context-Aware Semantic Inpainting

Haofeng Li, Guanbin Li+2 more|arXiv (Cornell University)|Dec 21, 2017

Generative Adversarial Networks and Image Synthesis32 references6 citations

TL;DR

This paper proposes Context-Aware Semantic Inpainting (CASI), a fully convolutional GAN-based method that preserves spatial structure and improves semantic consistency by using a context-aware loss with perceptual features. It outperforms state-of-the-art methods in both qualitative and quantitative evaluations, especially in edge preservation, texture realism, and semantic accuracy, as validated by novel metrics including local entropy error and semantic error.

ABSTRACT

Recently image inpainting has witnessed rapid progress due to generative adversarial networks (GAN) that are able to synthesize realistic contents. However, most existing GAN-based methods for semantic inpainting apply an auto-encoder architecture with a fully connected layer, which cannot accurately maintain spatial information. In addition, the discriminator in existing GANs struggle to understand high-level semantics within the image context and yield semantically consistent content. Existing evaluation criteria are biased towards blurry results and cannot well characterize edge preservation and visual authenticity in the inpainting results. In this paper, we propose an improved generative adversarial network to overcome the aforementioned limitations. Our proposed GAN-based framework consists of a fully convolutional design for the generator which helps to better preserve spatial structures and a joint loss function with a revised perceptual loss to capture high-level semantics in the context. Furthermore, we also introduce two novel measures to better assess the quality of image inpainting results. Experimental results demonstrate that our method outperforms the state of the art under a wide range of criteria.

Motivation & Objective

• To address the limitations of existing GAN-based semantic inpainting methods that use fully connected bottlenecks, which distort spatial information and fail to preserve structural and semantic consistency.
• To improve the discriminator's ability to assess high-level semantics by incorporating image context into the loss function, ensuring better alignment between synthesized content and surrounding context.
• To develop new quantitative metrics—local entropy error and semantic error—that better evaluate sharpness and semantic validity beyond traditional L2 and PSNR metrics.
• To achieve state-of-the-art performance in image inpainting by combining a fully convolutional generator with a joint loss function including perceptual loss.

Proposed method

• The generator uses a fully convolutional architecture without fully connected layers, preserving spatial structure and enabling variable input size processing.
• A context-aware loss is introduced by combining the synthesized region with its surrounding context and computing perceptual loss on the composite image using high-level features from a pre-trained network.
• The joint loss function includes a perceptual loss term that measures similarity between the composite image and ground truth in deep feature space, enhancing semantic consistency.
• A local entropy error is defined as the MSE or MAE of local entropy within a 9×9 neighborhood of the synthesized region, to penalize blurry results more accurately.
• A semantic error (SME) is computed as the hinge loss between the confidence scores of the ground truth and synthesized images from a pre-trained image classifier, measuring semantic correctness.
• The model is trained with a combination of adversarial loss, L2 loss, and perceptual loss, with hyperparameter λper tuned to balance perceptual and reconstruction fidelity.

Experimental results

Research questions

• RQ1 Can a fully convolutional generator architecture improve spatial structure preservation in semantic inpainting compared to auto-encoder models with fully connected bottlenecks?
• RQ2 Does incorporating the full image context into the discriminator's input improve structural and semantic consistency in generated regions?
• RQ3 Can perceptual loss computed on the composite image (synthesized region + context) better capture high-level semantics than standard perceptual loss on the synthesized region alone?
• RQ4 Can local entropy error serve as a more accurate metric than L2 or PSNR for detecting over-smoothed, blurry inpainting results?
• RQ5 Can semantic error, based on a pre-trained classifier’s confidence in the ground truth category, effectively measure the semantic validity of inpainted content?

Key findings

• CASI achieves the highest SSIM, FSIM, and FSIMc scores among all methods, indicating superior structural and visual quality in inpainted results.
• The LEMSE and LEMAE of CASI are the lowest across all methods, confirming its ability to produce sharper, less blurry results compared to baselines.
• CASI achieves the smallest semantic error (SME) when evaluated with both ResNet and VGG classifiers, demonstrating its capacity to restore semantically correct content.
• With λper = 0.4, CASI achieves the best balance in local entropy error, while λper = 0.7 performs best in semantic error and similarity metrics, indicating trade-offs between sharpness and semantics.
• The proposed local entropy error and semantic error metrics effectively detect over-smoothing and semantic inconsistencies, respectively, and correlate well with visual quality.
• CASI outperforms state-of-the-art methods on all evaluation criteria, including traditional metrics and the newly proposed ones, across diverse inpainting scenarios.