[论文解读] Diffusion-Based Scene Graph to Image Generation with Masked Contrastive Pre-Training

L. Yang, Zhilin Huang+7 more|arXiv (Cornell University)|Nov 21, 2022

Generative Adversarial Networks and Image Synthesis被引用 10

一句话总结

本文提出SGDiff，一种基于扩散的图像生成模型，通过使用掩码自编码和对比学习对场景图嵌入进行预训练，提升了场景图与图像之间的对齐效果。该方法在Visual Genome和COCO-Stuff数据集上优于当前最先进方法，在256×256图像生成任务中实现了SOTA的Inception Score（16.4）和FID（26.0）指标，同时通过场景图编辑实现了语义可控的图像生成。

ABSTRACT

Generating images from graph-structured inputs, such as scene graphs, is uniquely challenging due to the difficulty of aligning nodes and connections in graphs with objects and their relations in images. Most existing methods address this challenge by using scene layouts, which are image-like representations of scene graphs designed to capture the coarse structures of scene images. Because scene layouts are manually crafted, the alignment with images may not be fully optimized, causing suboptimal compliance between the generated images and the original scene graphs. To tackle this issue, we propose to learn scene graph embeddings by directly optimizing their alignment with images. Specifically, we pre-train an encoder to extract both global and local information from scene graphs that are predictive of the corresponding images, relying on two loss functions: masked autoencoding loss and contrastive loss. The former trains embeddings by reconstructing randomly masked image regions, while the latter trains embeddings to discriminate between compliant and non-compliant images according to the scene graph. Given these embeddings, we build a latent diffusion model to generate images from scene graphs. The resulting method, called SGDiff, allows for the semantic manipulation of generated images by modifying scene graph nodes and connections. On the Visual Genome and COCO-Stuff datasets, we demonstrate that SGDiff outperforms state-of-the-art methods, as measured by both the Inception Score and Fréchet Inception Distance (FID) metrics. We will release our source code and trained models at https://github.com/YangLing0818/SGDiff.

研究动机与目标

• 为解决由人工设计的中间表示（如场景布局）导致的场景图与生成图像之间的错位问题。
• 学习显式优化与对应图像对齐的场景图嵌入，以提升生成图像的保真度与结构合规性。
• 通过学习解耦且具有预测能力的嵌入，实现通过修改场景图来支持语义图像操控。
• 证明端到端学习的嵌入优于人工设计的场景布局表示，在定性和定量的图像生成基准测试中表现更优。

提出的方法

• 使用掩码自编码损失预训练场景图编码器，基于未掩码区域和图嵌入重建缺失的图像区域。
• 应用对比学习以区分符合场景图的图像-图对与不符合的图像-图对，增强全局结构对齐。
• 将局部（掩码自编码）和全局（对比）嵌入结合为统一表示，以提升预测能力。
• 将预训练的场景图嵌入作为条件输入，用于潜在扩散模型，生成高保真度且结构准确的图像。
• 利用学习到的嵌入通过修改场景图中的节点和关系，实现语义图像操控。
• 在Visual Genome和COCO-Stuff数据集上使用IS和FID指标进行模型训练与评估。

实验结果

研究问题

• RQ1 对场景图嵌入进行掩码对比预训练是否能提升图像生成质量与输入场景图的对齐程度？
• RQ2 局部（掩码自编码）与全局（对比）嵌入组件在图像生成性能中分别发挥何种作用？
• RQ3 端到端学习的场景图嵌入是否在场景图到图像生成任务中优于人工设计的场景布局表示？
• RQ4 学习到的嵌入是否能够通过场景图编辑支持细粒度的语义图像操控？

主要发现

• 掩码自编码与对比学习联合预训练在图像到图检索中达到73.4%的准确率，在图到图像检索中达到74.1%，优于单一损失方法。
• SGDiff在COCO-Stuff数据集上实现了16.4 ± 0.6的Inception Score和26.0的FID，优于one-hot（IS: 10.1, FID: 87.1）和基于布局的基线方法（IS: 12.3, FID: 52.7）。
• 掩码预训练提升了局部结构生成能力，使图像中物体细节更精细；对比预训练则增强了全局结构合规性。
• 消融实验表明，掩码自编码与对比损失均不可或缺，二者结合可获得最佳性能。
• 定性结果表明，SGDiff生成的图像比先前方法更真实且更符合场景图描述，尤其在复杂关系（如“吃”或“看着”）的生成上表现更优。