[Paper Review] TcGAN: Semantic-Aware and Structure-Preserved GANs with Individual Vision Transformer for Fast Arbitrary One-Shot Image Generation

Yunliang Jiang, Lili Yan+3 more|arXiv (Cornell University)|Feb 16, 2023

Image Processing Techniques and Applications4 citations

TL;DR

TcGAN proposes a semantic-aware and structure-preserving GAN for fast arbitrary one-shot image generation by integrating an individual Vision Transformer (ViT) to model long-range dependencies, overcoming CNNs' limited receptive fields. It achieves state-of-the-art performance with SIFID as low as 0.0117, LPIPS of 0.0560, and SSIM of 0.8449, while training in just ~10 minutes—1.6× faster than ConSinGAN and 6.5× faster than SinGAN.

ABSTRACT

One-shot image generation (OSG) with generative adversarial networks that learn from the internal patches of a given image has attracted world wide attention. In recent studies, scholars have primarily focused on extracting features of images from probabilistically distributed inputs with pure convolutional neural networks (CNNs). However, it is quite difficult for CNNs with limited receptive domain to extract and maintain the global structural information. Therefore, in this paper, we propose a novel structure-preserved method TcGAN with individual vision transformer to overcome the shortcomings of the existing one-shot image generation methods. Specifically, TcGAN preserves global structure of an image during training to be compatible with local details while maintaining the integrity of semantic-aware information by exploiting the powerful long-range dependencies modeling capability of the transformer. We also propose a new scaling formula having scale-invariance during the calculation period, which effectively improves the generated image quality of the OSG model on image super-resolution tasks. We present the design of the TcGAN converter framework, comprehensive experimental as well as ablation studies demonstrating the ability of TcGAN to achieve arbitrary image generation with the fastest running time. Lastly, TcGAN achieves the most excellent performance in terms of applying it to other image processing tasks, e.g., super-resolution as well as image harmonization, the results further prove its superiority.

Motivation & Objective

• Address the limitations of CNN-based one-shot image generation (OSG) methods, which struggle with global structure preservation and long-range dependency modeling.
• Overcome the poor generalization and structural inconsistencies in existing OSG models by leveraging the attention mechanism of Vision Transformers.
• Develop a scalable, fast-training framework for arbitrary image generation from a single image, suitable for real-world applications with limited data.
• Enable high-fidelity image super-resolution and image harmonization without additional labeling by leveraging scale-invariant scaling.
• Achieve faster training and superior image quality compared to state-of-the-art OSG methods like SinGAN, ConSinGAN, and PetsGAN.

Proposed method

• Introduce a novel TcGAN framework that combines a generator with an individual Vision Transformer (ViT) to model long-range dependencies and preserve global image structure.
• Design a scale-invariant scaling formula that maintains semantic consistency during image super-resolution, enabling high-fidelity generation without retraining or labeling.
• Replace the standard CNN-based feature extractor in OSG with a ViT-based encoder to enhance global context modeling and reduce structural artifacts.
• Implement a multi-scale, patch-based training strategy similar to SinGAN, but with ViT-enhanced feature learning to preserve both local details and global layout.
• Use a discriminator with self-attention to improve adversarial training stability and generate more realistic textures and structures.
• Integrate a TcGAN converter framework to enable zero-shot adaptation to downstream tasks like super-resolution and image harmonization.

Experimental results

Research questions

• RQ1 Can a Vision Transformer-based generator outperform CNN-based generators in preserving global structure and semantic consistency during one-shot image generation?
• RQ2 How does the proposed scale-invariant scaling formula improve image super-resolution quality without requiring additional labels?
• RQ3 What is the impact of integrating an individual ViT into the generator on training speed and inference quality in one-shot image generation?
• RQ4 Can TcGAN generalize effectively to downstream tasks such as image super-resolution and image harmonization without fine-tuning?
• RQ5 How does TcGAN compare in training efficiency and image quality to existing SOTA OSG methods like SinGAN, ConSinGAN, and PetsGAN?

Key findings

• TcGAN achieves the lowest SIFID score of 0.0117 on the Places50 dataset, indicating superior image quality and diversity compared to all baselines.
• On the CelebA50 dataset, TcGAN achieves an SSIM of 0.8882, demonstrating strong structural similarity to the source image while generating diverse samples.
• TcGAN reduces average training time to approximately 10 minutes, making it 1.6× faster than ConSinGAN, 2.8× faster than ExSinGAN, and 6.5× faster than SinGAN and SinDiffusion.
• The method achieves a LPIPS of 0.0560 on Places50, indicating high perceptual diversity and minimal redundancy in generated images.
• TcGAN generalizes effectively to super-resolution and image harmonization tasks, maintaining semantic consistency and structural fidelity without additional labeling.
• Ablation studies confirm that the ViT component is critical for global structure preservation, with performance dropping significantly when removed.