[Paper Review] SPG-Net: Segmentation Prediction and Guidance Network for Image Inpainting

Yuhang Song, Chao Yang+4 more|arXiv (Cornell University)|May 9, 2018

Generative Adversarial Networks and Image SynthesisComputer Science30 references118 citations

TL;DR

SPG-Net decomposes image inpainting into segmentation prediction (SP-Net) and segmentation-guided inpainting (SG-Net), leveraging segmentation maps to produce sharper boundaries and enabling interactive, multi-modal results. It outperforms prior methods on public datasets and supports user-guided edits.

ABSTRACT

In this paper, we focus on image inpainting task, aiming at recovering the missing area of an incomplete image given the context information. Recent development in deep generative models enables an efficient end-to-end framework for image synthesis and inpainting tasks, but existing methods based on generative models don't exploit the segmentation information to constrain the object shapes, which usually lead to blurry results on the boundary. To tackle this problem, we propose to introduce the semantic segmentation information, which disentangles the inter-class difference and intra-class variation for image inpainting. This leads to much clearer recovered boundary between semantically different regions and better texture within semantically consistent segments. Our model factorizes the image inpainting process into segmentation prediction (SP-Net) and segmentation guidance (SG-Net) as two steps, which predict the segmentation labels in the missing area first, and then generate segmentation guided inpainting results. Experiments on multiple public datasets show that our approach outperforms existing methods in optimizing the image inpainting quality, and the interactive segmentation guidance provides possibilities for multi-modal predictions of image inpainting.

Motivation & Objective

• Motivate the use of semantic segmentation to constrain object shapes in inpainting and reduce boundary blur.
• Propose a two-stage framework that first predicts segmentation in the hole and then guides image synthesis using that segmentation.
• Enable interactive editing of segmentation masks to produce multi-modal inpainting results.
• Demonstrate improved inpainting quality on public datasets and analyze the contributions via ablations.

Proposed method

• Split the inpainting pipeline into Segmentation Prediction Network (SP-Net) and Segmentation Guidance Network (SG-Net).
• SP-Net inputs incomplete image I0 and incomplete segmentation S0 to predict missing segmentation SR with a 4-down/4-up FCN-like generator and residual blocks, using a multi-scale GAN and a perceptual loss for realism.
• SG-Net takes I0 and the predicted full segmentation S to generate the final inpainted image I, using an architecture similar to SP-Net but with a tanh output and an additional AlexNet-based perceptual loss.
• Adversarial losses use three multi-scale PatchGAN discriminators to enforce global and local realism; perceptual loss aligns intermediate representations between the generated and ground-truth data (with mask-weighting).
• An AlexNet-based perceptual loss for SG-Net focuses on local hole patches with learned layer weights to improve perceptual similarity.

Experimental results

Research questions

• RQ1 Can segmentation guidance improve the realism and boundary sharpness of inpainted images compared to non-guided methods?
• RQ2 Does predicting segmentation in the missing hole help constrain plausible object layouts and textures for inpainting?
• RQ3 Is interactive editing of segmentation maps capable of producing multi-modal inpainting results?
• RQ4 How does SPG-Net perform on standard inpainting benchmarks relative to PatchMatch, GL, and GFC?

Key findings

• SPG-Net and SG-Net together produce sharper boundaries and better texture within semantically consistent regions than non-segmentation-guided methods.
• On Cityscapes, our method outperforms PatchMatch and GL on three of four quality metrics (ell_1, ell_2, SSIM, PSNR).
• Table 1 shows ell_1=641.3, ell_2=169.3, SSIM=0.9419, PSNR=30.34 for PatchMatch; ell_1=598.0, ell_2=94.78, SSIM=0.9576, PSNR=33.57 for GL; and ell_1=392.4, ell_2=98.95, SSIM=0.9591, PSNR=34.26 for Ours.
• A user study on Cityscapes reports our results preferred 70.8% of the time across 600 comparisons.
• Ablation shows baseline SG-Net without SP-Net yields blurrier boundaries, highlighting the benefit of segmentation-guided prediction.
• Interactive segmentation editing enables multi-modal inpainting outputs by guiding the hole content with alternative segmentation maps.