[Paper Review] Large Scale Image Completion via Co-Modulated Generative Adversarial Networks

Shengyu Zhao, Jonathan Cui+5 more|arXiv (Cornell University)|Mar 18, 2021

Generative Adversarial Networks and Image SynthesisComputer Science78 references135 citations

TL;DR

Introduces co-modulated GANs that jointly use conditional and stochastic style representations to enable high-quality, diverse large-scale image inpainting, and proposes a perceptual metric (P-IDS/U-IDS) for evaluation.

ABSTRACT

Numerous task-specific variants of conditional generative adversarial networks have been developed for image completion. Yet, a serious limitation remains that all existing algorithms tend to fail when handling large-scale missing regions. To overcome this challenge, we propose a generic new approach that bridges the gap between image-conditional and recent modulated unconditional generative architectures via co-modulation of both conditional and stochastic style representations. Also, due to the lack of good quantitative metrics for image completion, we propose the new Paired/Unpaired Inception Discriminative Score (P-IDS/U-IDS), which robustly measures the perceptual fidelity of inpainted images compared to real images via linear separability in a feature space. Experiments demonstrate superior performance in terms of both quality and diversity over state-of-the-art methods in free-form image completion and easy generalization to image-to-image translation. Code is available at https://github.com/zsyzzsoft/co-mod-gan.

Motivation & Objective

• Bridge the gap between image-conditional and unconditional modulated GANs to handle large-scale inpainting.
• Enable diverse and consistent inpaintings even with limited conditional information.
• Provide a robust quantitative metric for perceptual fidelity in image completion.
• Demonstrate generalization to image-to-image translation tasks.

Proposed method

• Propose co-modulation: jointly modulate features using both conditional encoder outputs and stochastic latent styles via a linear affine mapping.
• Leverage unconditional modulation architectures (StyleGAN2-style) within an image-conditional generator to retain stochasticity.
• Train with standard GAN losses without extra L1 losses to encourage diversity and realism.
• Introduce P-IDS/U-IDS: paired/unpaired Inception-based discriminative scores using a linear SVM in Inception feature space to measure perceptual fidelity.
• Demonstrate that co-modulation improves both quality and diversity, especially for large missing regions, and enables easy image-to-image translation.

Experimental results

Research questions

• RQ1 Can co-modulated GANs bridge image-conditioned and unconditional generators to handle large-scale image completion?
• RQ2 Do co-modulated models provide calibrated diversity and perceptual fidelity without extra supervision?
• RQ3 Is the proposed P-IDS/U-IDS metric robust, scalable, and correlated with human preferences?
• RQ4 Can the approach generalize to image-to-image translation tasks beyond inpainting?

Key findings

• Co-modulated GANs achieve superior quality and diversity in free-form image completion compared to state-of-the-art methods.
• Co-modulation preserves stochasticity and produces diverse outputs even with fixed input and mask.
• P-IDS/U-IDS metrics converge with small sample sizes and align well with human preferences, outperforming FID/KID in robustness and sensitivity.
• The method generalizes to image-to-image translation tasks such as edges-to-photos and COCO-Stuff label-to-image synthesis.
• Ablation studies show co-modulation outperforms vanilla and purely conditional modulation, especially for large missing regions.
• On FFHQ and Places2, the method achieves favorable quantitative results in both P-IDS/U-IDS and FID.