[Paper Review] You Only Need Adversarial Supervision for Semantic Image Synthesis

Vadim Sushko, Edgar Schönfeld+4 more|arXiv (Cornell University)|Dec 8, 2020

Generative Adversarial Networks and Image SynthesisComputer Science55 references70 citations

TL;DR

OASIS introduces a segmentation-based discriminator and 3D noise-driven generator to achieve high-quality, diverse semantic image synthesis using only adversarial supervision, eliminating the perceptual loss.

ABSTRACT

Despite their recent successes, GAN models for semantic image synthesis still suffer from poor image quality when trained with only adversarial supervision. Historically, additionally employing the VGG-based perceptual loss has helped to overcome this issue, significantly improving the synthesis quality, but at the same time limiting the progress of GAN models for semantic image synthesis. In this work, we propose a novel, simplified GAN model, which needs only adversarial supervision to achieve high quality results. We re-design the discriminator as a semantic segmentation network, directly using the given semantic label maps as the ground truth for training. By providing stronger supervision to the discriminator as well as to the generator through spatially- and semantically-aware discriminator feedback, we are able to synthesize images of higher fidelity with better alignment to their input label maps, making the use of the perceptual loss superfluous. Moreover, we enable high-quality multi-modal image synthesis through global and local sampling of a 3D noise tensor injected into the generator, which allows complete or partial image change. We show that images synthesized by our model are more diverse and follow the color and texture distributions of real images more closely. We achieve an average improvement of $6$ FID and $5$ mIoU points over the state of the art across different datasets using only adversarial supervision.

Motivation & Objective

• Motivate removing perceptual loss in semantic image synthesis by strengthening discriminator feedback.
• Design a discriminator that leverages semantic label maps for pixel-level, class-aware supervision.
• Develop a generator that enables multi-modal outputs via 3D noise injected at all layers.
• Demonstrate improved image quality and diversity on ADE20K, Cityscapes, and COCO-stuff compared to state of the art.

Proposed method

• Re-design the discriminator as a semantic segmentation network (N+1 classes: N real semantic classes + 1 fake) using inverse-frequency weighting for class balance.
• Introduce LabelMix regularization to encourage the discriminator to focus on semantic and structural differences, enforcing consistency under label-guided mixing.
• Replace the generator training with an adversarial loss that uses the segmentation-based discriminator.
• Enable multi-modal synthesis by injecting a 3D noise tensor into the generator at all layers, allowing global and local (per-segment/per-pixel) variation.
• Reduce complexity by removing an initial residual block to create a lighter generator (72M parameters).
• Compare label-map encoding strategies and ablate architectural choices to validate the discriminator’s effectiveness without perceptual loss.

Experimental results

Research questions

• RQ1 Can a segmentation-based discriminator provide stronger, semantically-aware feedback to the generator than traditional multi-scale discriminators?
• RQ2 Is perceptual (VGG) loss necessary for high-quality semantic image synthesis when the discriminator provides semantic-aware supervision?
• RQ3 Does 3D noise-based multi-modal synthesis improve diversity without sacrificing image fidelity?
• RQ4 How does LabelMix regularization affect the realism and semantic alignment of generated images?

Key findings

• OASIS achieves state-of-the-art results on ADE20K, Cityscapes, and COCO-stuff, averaging a 6-point FID improvement and a 5-point mIoU improvement over prior methods using only adversarial supervision.
• The segmentation-based discriminator (N+1 classes) provides per-pixel, semantically-aware feedback that replaces the need for perceptual losses like VGG.
• 3D noise-driven multi-modal synthesis enables global and local appearance changes, increasing diversity while maintaining semantic alignment.
• Ablations show replacing SPADE+ with the OASIS discriminator yields large FID/mIoU gains; adding 3D noise boosts diversity; the perceptual loss can slightly affect diversity and sometimes degrade FID without the better discriminator.
• LabelMix regularization improves pixel-level realism by encouraging the discriminator to respect semantic boundaries and content differences.
• Compared to SPADE+, OASIS without perceptual loss yields better FID and mIoU, demonstrating strong discriminator-driven supervision.