[Paper Review] MOGAN: Morphologic-structure-aware Generative Learning from a Single Image

Jinshu Chen, Qihui Xu+2 more|arXiv (Cornell University)|Mar 4, 2021

Generative Adversarial Networks and Image Synthesis60 references4 citations

TL;DR

MOGAN introduces a morphologic-structure-aware GAN that generates diverse, high-quality images from a single input by leveraging augmented data and a lightweight style injector to preserve object structures while enabling appearance variation. It achieves superior performance in ROI-based image generation, outperforming peers in both diversity and realism, as validated by quantitative metrics like SIFID and GQI.

ABSTRACT

In most interactive image generation tasks, given regions of interest (ROI) by users, the generated results are expected to have adequate diversities in appearance while maintaining correct and reasonable structures in original images. Such tasks become more challenging if only limited data is available. Recently proposed generative models complete training based on only one image. They pay much attention to the monolithic feature of the sample while ignoring the actual semantic information of different objects inside the sample. As a result, for ROI-based generation tasks, they may produce inappropriate samples with excessive randomicity and without maintaining the related objects' correct structures. To address this issue, this work introduces a MOrphologic-structure-aware Generative Adversarial Network named MOGAN that produces random samples with diverse appearances and reliable structures based on only one image. For training for ROI, we propose to utilize the data coming from the original image being augmented and bring in a novel module to transform such augmented data into knowledge containing both structures and appearances, thus enhancing the model's comprehension of the sample. To learn the rest areas other than ROI, we employ binary masks to ensure the generation isolated from ROI. Finally, we set parallel and hierarchical branches of the mentioned learning process. Compared with other single image GAN schemes, our approach focuses on internal features including the maintenance of rational structures and variation on appearance. Experiments confirm a better capacity of our model on ROI-based image generation tasks than its competitive peers.

Motivation & Objective

• Address the challenge of generating diverse, structurally accurate images from a single training sample, especially in ROI-based tasks.
• Overcome limitations of existing GANs that treat image patches uniformly, leading to structural distortions in generated samples.
• Improve sample quality by incorporating morphologic and structural knowledge from augmented versions of the single input image.
• Enable separate, parallel generation of regions of interest (ROI) and background using binary masks and specialized network branches.
• Achieve better balance between appearance diversity and structural fidelity than prior single-image GANs, particularly in interactive image generation tasks.

Proposed method

• Propose a parallel, hierarchical GAN architecture with separate branches for ROI and background generation, enabling isolated and targeted learning.
• Augment the original image into multiple forms (e.g., rotated, cropped, color-jittered) to synthesize diverse training data that preserves structural and morphological patterns.
• Introduce a lightweight style injector module that learns an affine transformation from augmented data and injects this knowledge into the generator to guide appearance variation while maintaining structure.
• Use binary masks to isolate ROI regions and guide the background branch to generate only the non-ROI areas, reducing generation complexity and improving fidelity.
• Integrate deformable convolutions and channel attention layers in the ROI branch to enhance feature representation and reduce noise and artifacts.
• Employ gated convolutions in the background branch to prevent the model from treating the mask as a semantic object, ensuring proper background generation.

Experimental results

Research questions

• RQ1 Can a GAN trained on a single image generate diverse, high-quality samples while preserving the structural integrity of objects in the input image?
• RQ2 How does the integration of morphologic-aware data augmentation and a style injector improve structural consistency and appearance diversity in single-image generation?
• RQ3 To what extent does separating ROI and background generation via parallel branches enhance the quality and realism of generated images?
• RQ4 How do individual components—such as the style injector, deformable convolutions, and gated convolutions—affect the model’s performance in terms of diversity and fidelity?
• RQ5 Can the proposed method outperform existing single-image GANs in both qualitative and quantitative metrics, especially under ROI-based generation tasks?

Key findings

• MOGAN achieves the lowest Single Image Frechet Inception Distance (SIFID) among all compared models when trained on the full image, indicating higher realism and reduced overfitting.
• When trained on ROI only, MOGAN maintains high diversity and low SIFID, outperforming SinGAN, ConSinGAN, and HP-VAE-GAN, which suffer from overfitting or chaotic generation.
• The Generation Quality Index (GQI), calculated as diversity divided by SIFID, is highest for MOGAN, demonstrating its optimal balance between diversity and quality.
• Ablation studies confirm that the style injector is critical—removing it leads to overfitting and reduced diversity, while deformable convolutions and channel attention improve structural stability.
• Gated convolutions in the background branch are essential to prevent the model from treating the mask as a semantic object, ensuring accurate background generation.
• Qualitative results show that MOGAN generates sharper edges and finer textures, especially in edited or ROI regions, compared to other models.