[Paper Review] PAI-Diffusion: Constructing and Serving a Family of Open Chinese Diffusion Models for Text-to-image Synthesis on the Cloud

Chengyu Wang, Zhongjie Duan+5 more|arXiv (Cornell University)|Sep 11, 2023

Generative Adversarial Networks and Image SynthesisComputer Science3 citations

TL;DR

PAI-Diffusion introduces a family of open-source Chinese diffusion models, including general and domain-specific variants, enhanced with LoRA and ControlNet for fine-grained image editing. It enables scalable, cloud-based text-to-image synthesis via integrated tools like a Chinese WebUI and diffusers-api, achieving 2–3× inference speedup using PAI-Blade optimization while maintaining high-quality outputs.

ABSTRACT

Text-to-image synthesis for the Chinese language poses unique challenges due to its large vocabulary size, and intricate character relationships. While existing diffusion models have shown promise in generating images from textual descriptions, they often neglect domain-specific contexts and lack robustness in handling the Chinese language. This paper introduces PAI-Diffusion, a comprehensive framework that addresses these limitations. PAI-Diffusion incorporates both general and domain-specific Chinese diffusion models, enabling the generation of contextually relevant images. It explores the potential of using LoRA and ControlNet for fine-grained image style transfer and image editing, empowering users with enhanced control over image generation. Moreover, PAI-Diffusion seamlessly integrates with Alibaba Cloud's Machine Learning Platform for AI, providing accessible and scalable solutions. All the Chinese diffusion model checkpoints, LoRAs, and ControlNets, including domain-specific ones, are publicly available. A user-friendly Chinese WebUI and the diffusers-api elastic inference toolkit, also open-sourced, further facilitate the easy deployment of PAI-Diffusion models in various environments, making it a valuable resource for Chinese text-to-image synthesis.

Motivation & Objective

• To address the challenges of Chinese text-to-image synthesis due to the language's large vocabulary and complex character dependencies.
• To develop a comprehensive framework that supports both general and domain-specific image generation for Chinese contexts, such as poetry, cuisine, and traditional art.
• To enable fine-grained control over image generation through LoRA and ControlNet integration for style transfer and editing.
• To provide scalable, cloud-native deployment via Alibaba Cloud’s PAI platform and open-source toolkits for broad accessibility.
• To release all models, LoRAs, ControlNets, and toolkits under open licenses to foster community collaboration and innovation.

Proposed method

• The framework employs a family of diffusion models pre-trained on diverse Chinese text-image datasets, with specialized variants for domains like Chinese cuisine and classical poetry.
• LoRA (Low-Rank Adaptation) is applied to enable efficient fine-tuning and style transfer, allowing users to modify visual attributes without retraining the full model.
• ControlNet is integrated to guide image generation based on conditions such as pose, depth, or segmentation maps, enabling precise editing and layout control.
• The diffusers-api toolkit exposes the models as RESTful web services, supporting multiple inference tasks including text-to-image, image-to-image, and in-painting.
• PAI-Blade, an AI compiler, is used to optimize inference, reducing end-to-end latency and GPU memory usage by up to 18% compared to native PyTorch.
• A customized Chinese WebUI extension enhances the Stable Diffusion WebUI for better Chinese prompt handling and user experience.

Experimental results

Research questions

• RQ1 How can diffusion models be effectively adapted to handle the linguistic complexity of the Chinese language, including its large vocabulary and character-level dependencies?
• RQ2 To what extent can domain-specific fine-tuning improve the relevance and quality of generated images for niche Chinese cultural contexts?
• RQ3 Can LoRA and ControlNet be effectively leveraged to enable fine-grained control over image style and composition in Chinese text-to-image generation?
• RQ4 How does cloud-based deployment with optimized inference impact scalability, performance, and accessibility for Chinese diffusion models?
• RQ5 What is the performance gain of using a compiler-optimized inference pipeline (PAI-Blade) over standard PyTorch inference in real-world deployment scenarios?

Key findings

• The diffusers-api toolkit achieved a 2.96-second average inference time on an NVIDIA A10 GPU, representing a 2.14× speedup over the native PyTorch implementation (6.34 seconds).
• GPU memory consumption was reduced from 6.94 GB to 5.56 GB when using the optimized diffusers-api pipeline, indicating improved resource efficiency.
• The framework successfully supports diverse domain-specific models, including those for Chinese cuisine, poetry, and traditional paintings, enabling contextually relevant image generation.
• LoRA and ControlNet were effectively deployed with Chinese models, enabling users to manipulate image styles and structures with high precision.
• The open-sourcing of all model checkpoints, LoRAs, ControlNets, and toolkits (WebUI and diffusers-api) enables widespread adoption and customization across diverse environments.
• Performance benchmarks confirmed that PAI-Blade optimization maintains full precision while significantly reducing latency and memory usage in cloud-based inference.