[Paper Review] Interactive Data Synthesis for Systematic Vision Adaptation via LLMs-AIGCs Collaboration

Qifan Yu, Juncheng Li+3 more|arXiv (Cornell University)|May 22, 2023

Multimodal Machine Learning Applications4 citations

TL;DR

This paper proposes ChatGenImage, a novel interactive data synthesis framework that leverages collaborative prompting between large language models (LLMs) and AIGC models (e.g., Stable Diffusion) to generate high-quality, diverse, and finely annotated synthetic images for systematic vision adaptation. By iteratively refining image generation through LLM-guided global and local prompts, the method enables controllable, cost-effective data augmentation, especially for rare or unseen domains, significantly improving downstream model generalization with minimal human effort.

ABSTRACT

Recent text-to-image generation models have shown promising results in generating high-fidelity photo-realistic images. In parallel, the problem of data scarcity has brought a growing interest in employing AIGC technology for high-quality data expansion. However, this paradigm requires well-designed prompt engineering that cost-less data expansion and labeling remain under-explored. Inspired by LLM's powerful capability in task guidance, we propose a new paradigm of annotated data expansion named as ChatGenImage. The core idea behind it is to leverage the complementary strengths of diverse models to establish a highly effective and user-friendly pipeline for interactive data augmentation. In this work, we extensively study how LLMs communicate with AIGC model to achieve more controllable image generation and make the first attempt to collaborate them for automatic data augmentation for a variety of downstream tasks. Finally, we present fascinating results obtained from our ChatGenImage framework and demonstrate the powerful potential of our synthetic data for systematic vision adaptation. Our codes are available at https://github.com/Yuqifan1117/Labal-Anything-Pipeline.

Motivation & Objective

• Address data scarcity and domain shift in vision tasks by enabling scalable, low-cost data augmentation using synthetic data.
• Overcome limitations of existing AIGC-based data synthesis, which often produce simple, object-centric images with limited scene complexity and domain diversity.
• Leverage the zero-shot reasoning and instruction-following capabilities of LLMs to guide AIGC models toward generating complex, semantically rich scenes with fine-grained annotations.
• Develop an interactive, iterative pipeline that combines LLMs for prompt engineering and AIGC for image generation, enabling controllable and scalable data expansion for downstream vision tasks.

Proposed method

• Employ gpt-3.5-turbo as the LLM to generate global and local prompts based on input labels, using a zero-shot prompting strategy to guide image generation.
• Use Stable Diffusion v1.5 as the AIGC backbone to generate 512×512 resolution images from LLM-generated prompts, with temperature set to 0 for inference stability.
• Implement an iterative refinement loop where the LLM analyzes generated images, identifies missing or incorrect elements, and issues targeted local editing prompts to improve realism and semantic accuracy.
• Integrate a label foundation toolkit to automatically detect and verify object presence and spatial relationships in generated images, ensuring semantic consistency.
• Apply open-vocabulary object detection to filter images that fail to preserve original object semantics or background context, retaining only high-quality, semantically aligned outputs.
• Rank image candidates using a semantic alignment score between textual constraints and image content, discarding low-confidence results to ensure data quality.

Experimental results

Research questions

• RQ1 Can LLMs effectively guide AIGC models to generate complex, diverse, and semantically accurate images through iterative prompting, even for rare or unfamiliar concepts?
• RQ2 How does the collaboration between LLMs and AIGC models improve the controllability and quality of synthetic data compared to naive prompt-based AIGC generation?
• RQ3 To what extent can the synthetic data generated via this interactive pipeline enhance the generalization and robustness of downstream vision models in unseen domains?
• RQ4 Can the framework systematically generate fine-grained annotations and diverse scene compositions (e.g., night, snow, forest) that are often neglected in standard data augmentation?
• RQ5 How effective is the iterative refinement process in correcting hallucinations and improving object consistency and spatial relationships in generated images?

Key findings

• ChatGenImage successfully generates high-fidelity, complex scene images with fine-grained annotations for rare and endangered species, even when such concepts are unfamiliar to standard classifiers.
• The iterative LLM-AIGC collaboration significantly improves image quality and semantic accuracy, enabling the generation of diverse scenes (e.g., mountain, forest, night) that are often underrepresented in standard data augmentation.
• The framework achieves high semantic alignment between textual prompts and generated images, with filtering based on object detection and semantic scoring ensuring only high-quality outputs are retained.
• The method enables cost-effective, scalable data augmentation by minimizing human intervention, with the entire pipeline running on a single consumer-grade GPU (Nvidia RTX 3090).
• Qualitative results demonstrate that the LLM can effectively guide AIGC models to incorporate specific object relations and spatial layouts, such as snow trees and streams in mountainous scenes.
• The framework shows strong potential for systematic vision adaptation by generating synthetic data that covers unseen domains and long-tail categories, enhancing model robustness in real-world deployment.