[Paper Review] HiCo: Hierarchical Controllable Diffusion Model for Layout-to-image Generation

Bo Cheng, Yuhang Ma+6 more|arXiv (Cornell University)|Oct 18, 2024

Computer Graphics and Visualization Techniques4 citations

TL;DR

HiCo proposes a hierarchical controllable diffusion model for layout-to-image generation that uses a multi-branch, weight-shared conditioning structure to achieve spatial disentanglement, improving object consistency and image quality in complex layouts. It achieves state-of-the-art performance on both the new HiCo-7K benchmark and COCO-3K, with faster inference and enhanced controllability via FuseNet fusion and support for LoRA/LCM integration.

ABSTRACT

The task of layout-to-image generation involves synthesizing images based on the captions of objects and their spatial positions. Existing methods still struggle in complex layout generation, where common bad cases include object missing, inconsistent lighting, conflicting view angles, etc. To effectively address these issues, we propose a extbf{Hi}erarchical extbf{Co}ntrollable (HiCo) diffusion model for layout-to-image generation, featuring object seperable conditioning branch structure. Our key insight is to achieve spatial disentanglement through hierarchical modeling of layouts. We use a multi branch structure to represent hierarchy and aggregate them in fusion module. To evaluate the performance of multi-objective controllable layout generation in natural scenes, we introduce the HiCo-7K benchmark, derived from the GRIT-20M dataset and manually cleaned. https://github.com/360CVGroup/HiCo_T2I.

Motivation & Objective

• To address persistent issues in layout-to-image generation, such as object missing, inconsistent lighting, and conflicting view angles in complex scenes.
• To improve spatial disentanglement and compositional harmony by modeling layouts hierarchically through multiple conditioning branches.
• To introduce a new, high-quality benchmark, HiCo-7K, for evaluating multi-objective, controllable layout generation in natural scenes.
• To enable flexible scalability through integration with LoRA, LCM, and model checkpoint switching.
• To demonstrate superior performance in both closed-set and open-ended layout generation tasks.

Proposed method

• The model employs a multi-branch, weight-shared architecture to separately extract hierarchical layout features for each object, enabling localized control.
• A FuseNet module non-parametrically aggregates features from individual object branches and a global background branch using sum, average, or attention-based masking.
• The global background branch uses a global prompt to guide overall scene coherence and occlusion ordering.
• The method integrates with pre-trained text-to-image diffusion models (e.g., Stable Diffusion) via a conditioning branch structure similar to ControlNet or IP-Adapter.
• Inference supports both parallel and serial modes, with parallel mode enabling batch processing across object branches for speedup.
• The model is compatible with LoRA fine-tuning and LCM (Latent Consistency Models) for fast, high-quality inference.

Experimental results

Research questions

• RQ1 Can hierarchical modeling of layout features improve object consistency and spatial coherence in complex layout-to-image generation?
• RQ2 How does the proposed multi-branch, fused conditioning structure compare to existing training-free and training-based methods in handling complex layouts?
• RQ3 To what extent can the HiCo model maintain high image quality and controllability while supporting fast inference and model flexibility?
• RQ4 How effective is the new HiCo-7K benchmark in objectively evaluating layout-to-image generation in natural scenes?
• RQ5 Can the model effectively handle open-ended, fine-grained textual descriptions and multi-concept generation using LoRA?

Key findings

• HiCo achieves state-of-the-art performance on the open-ended HiCo-7K benchmark, with a Fréchet Inception Distance (FID) of 15.26 and an Average Recall (AR) of 39.51.
• On the closed-set COCO-3K dataset, HiCo achieves an FID of 18.78 and an AR of 36.30, outperforming existing methods like CAG and GLIGEN.
• The FuseNet module with mask-based fusion yields the best results, reducing FID to 15.26 and increasing LocalIoU Score to 59.07 on HiCo-7K.
• HiCo achieves the fastest inference time and second-lowest GPU memory usage among six compared models on a 24GB V100 GPU, with a 512×512 image generation time under 1 second.
• The model supports fast inference via LCM and Lightning inference modes, enabling real-time generation with high quality.
• Subjective evaluation confirms that HiCo generates more harmonious and realistic images than baselines, especially in complex layouts with overlapping objects.