[Paper Review] Semantically Multi-modal Image Synthesis

Zhen Zhu, Zhiliang Xu+2 more|arXiv (Cornell University)|Mar 28, 2020

Generative Adversarial Networks and Image Synthesis59 references4 citations

TL;DR

This paper proposes GroupDNet, a novel generator architecture for semantically multi-modal image synthesis (SMIS) that uses group convolutions with decreasing group numbers in the decoder to enable fine-grained, controllable generation of diverse images from semantic labels. The method achieves state-of-the-art image quality on DeepFashion and Cityscapes, and superior controllability over semantic parts while maintaining high fidelity, especially on large-class datasets like ADE20K.

ABSTRACT

In this paper, we focus on semantically multi-modal image synthesis (SMIS) task, namely, generating multi-modal images at the semantic level. Previous work seeks to use multiple class-specific generators, constraining its usage in datasets with a small number of classes. We instead propose a novel Group Decreasing Network (GroupDNet) that leverages group convolutions in the generator and progressively decreases the group numbers of the convolutions in the decoder. Consequently, GroupDNet is armed with much more controllability on translating semantic labels to natural images and has plausible high-quality yields for datasets with many classes. Experiments on several challenging datasets demonstrate the superiority of GroupDNet on performing the SMIS task. We also show that GroupDNet is capable of performing a wide range of interesting synthesis applications. Codes and models are available at: https://github.com/Seanseattle/SMIS.

Motivation & Objective

• To address the limitation of existing methods in generating diverse, controllable images from semantic labels, especially when multiple classes are involved.
• To overcome the inefficiency and scalability issues of multiple class-specific generators in multi-modal image synthesis.
• To enable user-controlled, semantic-level editing—changing one part without affecting others—while preserving image quality and consistency.
• To improve model efficiency and scalability for datasets with a large number of semantic classes, such as ADE20K.
• To develop a unified generative model that supports diverse applications like appearance mixing, semantic manipulation, and style morphing.

Proposed method

• Replacing all standard convolutions in the generator with group convolutions to enable class-specific feature learning and control.
• Progressively decreasing the number of groups in the decoder convolutions to model inter-class correlations (e.g., color consistency between grass and leaves).
• Using a unified generator with group convolutional layers that allow independent control over different semantic parts during inference.
• Leveraging an encoder to extract style codes from real images, enabling style transfer and morphing applications.
• Training the model using a GAN-based framework with adversarial loss, L1 loss, and perceptual loss to ensure realism and fidelity.
• Integrating the GroupDNet generator into a conditional image synthesis pipeline, similar to SPADE, for consistent training and evaluation.

Experimental results

Research questions

• RQ1 Can a unified generator with group convolutions and decreasing group numbers achieve better controllability in multi-modal image synthesis than multiple class-specific generators?
• RQ2 How does the progressive reduction of group numbers in the decoder affect inter-class correlation modeling and image quality?
• RQ3 To what extent can GroupDNet maintain high image quality while enabling fine-grained, semantic-level control over individual image parts?
• RQ4 Can GroupDNet generalize to diverse applications such as appearance mixing, semantic manipulation, and style morphing?
• RQ5 How does GroupDNet compare to state-of-the-art methods in terms of FID, mIoU, and accuracy on large-scale datasets like ADE20K?

Key findings

• GroupDNet achieves state-of-the-art FID scores on DeepFashion (9.50) and Cityscapes (49.81), matching or exceeding SPADE and other SOTA methods.
• On the ADE20K dataset, GroupDNet achieves a mIoU of 30.4 and accuracy of 77.1, outperforming all baselines despite the high class count.
• The model demonstrates superior controllability: changing one semantic part (e.g., clothing) does not alter other parts, as validated in qualitative results.
• GroupDNet enables novel applications such as appearance mixture (thousands of unique person images from a single mask), semantic manipulation (e.g., inserting a bed), and style morphing.
• The method reduces training and inference costs compared to multiple generator approaches, with inference speed nearly double that of multi-generator baselines.
• The proposed mCSD and mOCD metrics confirm that GroupDNet generates high diversity in target classes while maintaining low diversity in other classes, indicating precise control.