Skip to main content
QUICK REVIEW

[论文解读] Composer: Creative and Controllable Image Synthesis with Composable Conditions

Lianghua Huang, Di Chen|arXiv (Cornell University)|Feb 20, 2023
Generative Adversarial Networks and Image SynthesisComputer Science被引用 52
一句话总结

Composer 引入了一种可组合的扩散框架,将图像分解为灵活的表示,并在推理阶段根据多种、用户可选择的条件重新配置它们,从而实现高度可控的图像合成与操控。它在不重新训练的情况下,在多项任务上实现了高质量。

ABSTRACT

Recent large-scale generative models learned on big data are capable of synthesizing incredible images yet suffer from limited controllability. This work offers a new generation paradigm that allows flexible control of the output image, such as spatial layout and palette, while maintaining the synthesis quality and model creativity. With compositionality as the core idea, we first decompose an image into representative factors, and then train a diffusion model with all these factors as the conditions to recompose the input. At the inference stage, the rich intermediate representations work as composable elements, leading to a huge design space (i.e., exponentially proportional to the number of decomposed factors) for customizable content creation. It is noteworthy that our approach, which we call Composer, supports various levels of conditions, such as text description as the global information, depth map and sketch as the local guidance, color histogram for low-level details, etc. Besides improving controllability, we confirm that Composer serves as a general framework and facilitates a wide range of classical generative tasks without retraining. Code and models will be made available.

研究动机与目标

  • 证明在多条件下的可组合性如何扩展图像合成的可控设计空间。
  • 开发一个具有分解–组合流程的多条件扩散模型。
  • 表明该框架在不重新训练的情况下支持多样化任务(如文本到图像、风格迁移、姿态迁移等)。
  • 提供区域可控编辑和鲁棒的高分辨率上采样机制。

提出的方法

  • 在训练期间实时提取,将每张图像分解为八种表示(描述、语义/风格、颜色直方图、草图、实例、深度图、强度、遮罩)。
  • 训练一个多条件扩散模型(UNet 主干)以从这些表示的子集重建图像。
  • 对高级别表示使用全局条件,对在空间上变化的表示使用局部条件,结合加性嵌入和跨注意力引导。
  • 采用无分类器引导(classifier-free)和双向引导,以实现灵活的操作和解耦的编辑。
  • 使用两种无条件扩散上采样模型将生成的64x64图像放大至256x256再到1024x1024。
  • 可选地使用先验模型将描述投影到图像嵌入,并采用按条件随机丢弃的训练策略以鼓励从多样的条件组合中学习。
Figure 1: Concept of compositional image synthesis , which first decomposes an image to a set of basic components and then recomposes a new one with high creativity and controllability. To this end, the components in various formats serve as conditions in the generation process and allow flexible cu
Figure 1: Concept of compositional image synthesis , which first decomposes an image to a set of basic components and then recomposes a new one with high creativity and controllability. To this end, the components in various formats serve as conditions in the generation process and allow flexible cu

实验结果

研究问题

  • RQ1可组合条件是否能将图像合成的可控空间扩展到超越传统的单一条件模型?
  • RQ2在不重新训练的情况下,如何从多样、潜在不兼容的表示中重建并操纵图像?
  • RQ3不同条件模态(全局 vs 局部)对生成质量和可控性的影响是什么?
  • RQ4该框架能否在单一模型内重新表述传统的生成任务(如上色、风格迁移、翻译、编辑)?

主要发现

  • 在仅使用描述作为条件时,Composer 在 COCO 数据集上实现了有竞争力的文本到图像质量,零-shot FID 为 9.2,CLIP 分数为 0.28。
  • 该方法在无需重新训练的情况下实现了广泛的任务和操作,包括通过草图、深度图或调色板进行重新配置。
  • 通过使用遮罩条件,可以进行区域特定的编辑,使修改局限于用户定义的区域。
  • 通过遍历或选择性地组合全局表示,可以实现插值和变化,产生多种可控方向。
  • 通过 dropout 的多条件联合训练使模型从多样的条件组合中学习,而上采样模型则提供高分辨率输出(高达 1024x1024)。
  • 该方法在传统生成任务和新颖的组合情境中实现了高度的可控性和灵活性。
Figure 2: (a) Image variation. For each example, the first column shows the source image, while the subsequent four columns are variations of the source image produced by conditioning Composer on different subsets of its representations. (b) Image interpolation. On the first row are the results of i
Figure 2: (a) Image variation. For each example, the first column shows the source image, while the subsequent four columns are variations of the source image produced by conditioning Composer on different subsets of its representations. (b) Image interpolation. On the first row are the results of i

阅读本论文

在 Nubint 阅读器中打开原文,边读边就任意段落提问。

在 Nubint 中阅读本论文

本解读由 AI 根据论文原文生成,引用前请核对原文。