[论文解读] SPG-Net: Segmentation Prediction and Guidance Network for Image Inpainting

Yuhang Song, Chao Yang+4 more|arXiv (Cornell University)|May 9, 2018

Generative Adversarial Networks and Image Synthesis参考文献 30被引用 118

一句话总结

SPG-Net 将图像修复分解为分割预测（SP-Net）与分割引导的修复（SG-Net），利用分割图以产生更清晰的边界并实现交互式、多模态结果。它在公开数据集上优于此前方法，并支持用户引导的编辑。

ABSTRACT

In this paper, we focus on image inpainting task, aiming at recovering the missing area of an incomplete image given the context information. Recent development in deep generative models enables an efficient end-to-end framework for image synthesis and inpainting tasks, but existing methods based on generative models don't exploit the segmentation information to constrain the object shapes, which usually lead to blurry results on the boundary. To tackle this problem, we propose to introduce the semantic segmentation information, which disentangles the inter-class difference and intra-class variation for image inpainting. This leads to much clearer recovered boundary between semantically different regions and better texture within semantically consistent segments. Our model factorizes the image inpainting process into segmentation prediction (SP-Net) and segmentation guidance (SG-Net) as two steps, which predict the segmentation labels in the missing area first, and then generate segmentation guided inpainting results. Experiments on multiple public datasets show that our approach outperforms existing methods in optimizing the image inpainting quality, and the interactive segmentation guidance provides possibilities for multi-modal predictions of image inpainting.

研究动机与目标

• Motivate the use of semantic segmentation to constrain object shapes in inpainting and reduce boundary blur.
• Propose a two-stage framework that first predicts segmentation in the hole and then guides image synthesis using that segmentation.
• Enable interactive editing of segmentation masks to produce multi-modal inpainting results.
• Demonstrate improved inpainting quality on public datasets and analyze the contributions via ablations.

提出的方法

• Split the inpainting pipeline into Segmentation Prediction Network (SP-Net) and Segmentation Guidance Network (SG-Net).
• SP-Net inputs incomplete image I0 and incomplete segmentation S0 to predict missing segmentation SR with a 4-down/4-up FCN-like generator and residual blocks, using a multi-scale GAN and a perceptual loss for realism.
• SG-Net takes I0 and the predicted full segmentation S to generate the final inpainted image I, using an architecture similar to SP-Net but with a tanh output and an additional AlexNet-based perceptual loss.
• Adversarial losses use three multi-scale PatchGAN discriminators to enforce global and local realism; perceptual loss aligns intermediate representations between the generated and ground-truth data (with mask-weighting).
• An AlexNet-based perceptual loss for SG-Net focuses on local hole patches with learned layer weights to improve perceptual similarity.]
• research_questions':['Can segmentation guidance improve the realism and boundary sharpness of inpainted images compared to non-guided methods?','Does predicting segmentation in the missing hole help constrain plausible object layouts and textures for inpainting?','Is interactive editing of segmentation maps capable of producing multi-modal inpainting results?','How does SPG-Net perform on standard inpainting benchmarks relative to PatchMatch, GL, and GFC?'],
• key_findings':['SPG-Net and SG-Net together produce sharper boundaries and better texture within semantically consistent regions than non-segmentation-guided methods.','On Cityscapes, our method outperforms PatchMatch and GL on three of four quality metrics (ell_1, ell_2, SSIM, PSNR).','Table 1 shows ell_1=641.3, ell_2=169.3, SSIM=0.9419, PSNR=30.34 for PatchMatch; ell_1=598.0, ell_2=94.78, SSIM=0.9576, PSNR=33.57 for GL; and ell_1=392.4, ell_2=98.95, SSIM=0.9591, PSNR=34.26 for Ours.','A user study on Cityscapes reports our results preferred 70.8% of the time across 600 comparisons.','Ablation shows baseline SG-Net without SP-Net yields blurrier boundaries, highlighting the benefit of segmentation-guided prediction.','Interactive segmentation editing enables multi-modal inpainting outputs by guiding the hole content with alternative segmentation maps.']
• table_headers':['方法','ell_1','ell_2','SSIM','PSNR'],"table_rows":[["PatchMatch","641.3","169.3","0.9419","30.34"],["GL","598.0","94.78","0.9576","33.57"],["Ours","392.4","98.95","0.9591","34.26"]]} }`}json``-style`?Oops`Let's provide clean JSON object instead.}
• table_rows_TRANSLATED_PLACEHOLDER

实验结果

研究问题

• RQ1 Can segmentation guidance improve the realism and boundary sharpness of inpainted images compared to non-guided methods?
• RQ2 Does predicting segmentation in the missing hole help constrain plausible object layouts and textures for inpainting?
• RQ3 Is interactive editing of segmentation maps capable of producing multi-modal inpainting results?
• RQ4 How does SPG-Net perform on standard inpainting benchmarks relative to PatchMatch, GL, and GFC?

主要发现

方法	ell_1	ell_2	SSIM	PSNR
PatchMatch	641.3	169.3	0.9419	30.34
GL	598.0	94.78	0.9576	33.57
Ours	392.4	98.95	0.9591	34.26

• SPG-Net and SG-Net together produce sharper boundaries and better texture within semantically consistent regions than non-segmentation-guided methods.
• On Cityscapes, our method outperforms PatchMatch and GL on three of four quality metrics (ell_1, ell_2, SSIM, PSNR).
• Table 1 shows ell_1=641.3, ell_2=169.3, SSIM=0.9419, PSNR=30.34 for PatchMatch; ell_1=598.0, ell_2=94.78, SSIM=0.9576, PSNR=33.57 for GL; and ell_1=392.4, ell_2=98.95, SSIM=0.9591, PSNR=34.26 for Ours.
• A user study on Cityscapes reports our results preferred 70.8% of the time across 600 comparisons.
• Ablation shows baseline SG-Net without SP-Net yields blurrier boundaries, highlighting the benefit of segmentation-guided prediction.
• Interactive segmentation editing enables multi-modal inpainting outputs by guiding the hole content with alternative segmentation maps.