[论文解读] SimGraph: A Unified Framework for Scene Graph-Based Image Generation and Editing

Thanh-Nhan Vo, Trong-Thuan Nguyen+2 more|arXiv (Cornell University)|Jan 29, 2026

Generative Adversarial Networks and Image Synthesis被引用 0

一句话总结

SimGraph 将基于场景图的图像生成与编辑统一到同一模型中，通过将基于标记的生成与受场景图引导的扩散式编辑相结合，提升保真度与效率。

ABSTRACT

Recent advancements in Generative Artificial Intelligence (GenAI) have significantly enhanced the capabilities of both image generation and editing. However, current approaches often treat these tasks separately, leading to inefficiencies and challenges in maintaining spatial consistency and semantic coherence between generated content and edits. Moreover, a major obstacle is the lack of structured control over object relationships and spatial arrangements. Scene graph-based methods, which represent objects and their interrelationships in a structured format, offer a solution by providing greater control over composition and interactions in both image generation and editing. To address this, we introduce SimGraph, a unified framework that integrates scene graph-based image generation and editing, enabling precise control over object interactions, layouts, and spatial coherence. In particular, our framework integrates token-based generation and diffusion-based editing within a single scene graph-driven model, ensuring high-quality and consistent results. Through extensive experiments, we empirically demonstrate that our approach outperforms existing state-of-the-art methods.

研究动机与目标

• 需要一个能同时实现生成/编辑且在复杂场景中保持空间与语义一致性的框架。
• 引入一个统一模型，使用场景图来控制生成与编辑路径。
• 利用基于标记的生成与基于扩散的编辑，条件化自场景图派生的字幕与提示。
• 展示在保真度、效率与语义对齐方面相较现有基于场景图的方法的改进。

提出的方法

• 使用多语言LLM基提取器（如 Qwen-VL）从输入图像中提取场景图。
• 从场景图构建字幕 C，以条件化生成和编辑流水线。
• 通过一个条件于 CLIP 文本嵌入的 Visual AutoRegressive (VAR) 模型执行基于标记的图像生成。
• 实现基于扩散的图像编辑，采用联合源/目标条件以在应用编辑的同时保留背景（LEDIT++-based 方法）。
• 从场景图变更创建源/目标提示（Alg. 1），以将保留内容与编辑内容分离（Alg. 2）。
• 以覆盖 token-based 生成（C）和扩散式编辑条件的条件负对数似然目标进行训练。

Figure 1 : Illustration of SimGraph, which shares the same strategy for scene graph extraction using MLLM (e.g., Qwen-VL [ 1 ] ) (introduced in Sec. 4.1 ). In addition, our framework simultaneously integrates token-based image generation (introduced in Sec. 4.2 ) and diffusion model for image editin

实验结果

研究问题

• RQ1 如何在单一的基于场景图的框架下统一生成与编辑？
• RQ2 基于场景图派生的字幕与提示能否为生成与编辑任务提供鲁棒、连贯的控制？
• RQ3 在场景图条件下，基于 token 的生成与扩散式编辑是否比现有方法具有更高的保真度和语义对齐？
• RQ4 将生成与编辑整合为一个流水线时，能实现哪些效率提升？

主要发现

Model	Accuracy	Fidelity
SG2IM	0.11	0.57
DiffSG	0.01	0.13
SimGraph (Ours)	0.32	0.87

• SimGraph 在保真度更高、准确性具有竞争力且效率提升的前提下，相较现有基于场景图的编辑方法表现更优。
• 在 EditVal 上，SimGraph 达到准确率 0.32 与保真度 0.87，保真度优于 SGEdit 和 DiffSG。
• 定性结果显示：能从场景图实现连贯的生成，并实现文本引导的精确编辑，且能保留上下文。
• 通过将生成与编辑统一到一个通道并进行联合条件化，显著降低计算开销。
• 每张图像的编辑耗时约 20-30 秒，明显快于某些先前方法。

Figure 2 : Illustration of image generation and editing from scene graphs using our framework. The left side shows the input image with its corresponding scene graph. The generated image on the right demonstrates the model’s ability to faithfully recreate the scene from the extracted scene graph. Th