[Paper Review] Image Translation as Diffusion Visual Programmers

Cheng Han, James C. Liang+6 more|arXiv (Cornell University)|Jan 18, 2024

Cell Image Analysis Techniques10 citations

TL;DR

DVP combines a condition-flexible diffusion model with GPT-driven visual programming to perform controllable, explainable image translation by decomposing tasks into RoI identification, editing, and localization. It achieves robust, high-fidelity translations without hand-tuned guidance scales.

ABSTRACT

We introduce the novel Diffusion Visual Programmer (DVP), a neuro-symbolic image translation framework. Our proposed DVP seamlessly embeds a condition-flexible diffusion model within the GPT architecture, orchestrating a coherent sequence of visual programs (i.e., computer vision models) for various pro-symbolic steps, which span RoI identification, style transfer, and position manipulation, facilitating transparent and controllable image translation processes. Extensive experiments demonstrate DVP's remarkable performance, surpassing concurrent arts. This success can be attributed to several key features of DVP: First, DVP achieves condition-flexible translation via instance normalization, enabling the model to eliminate sensitivity caused by the manual guidance and optimally focus on textual descriptions for high-quality content generation. Second, the framework enhances in-context reasoning by deciphering intricate high-dimensional concepts in feature spaces into more accessible low-dimensional symbols (e.g., [Prompt], [RoI object]), allowing for localized, context-free editing while maintaining overall coherence. Last but not least, DVP improves systemic controllability and explainability by offering explicit symbolic representations at each programming stage, empowering users to intuitively interpret and modify results. Our research marks a substantial step towards harmonizing artificial image translation processes with cognitive intelligence, promising broader applications.

Motivation & Objective

• Translate images by locating Regions of Interest (RoIs) and applying targeted style/content changes while preserving context.
• Introduce a condition-flexible diffusion model that reduces reliance on manual guidance scales.
• Enable in-context reasoning through visual programming to decompose high-dimensional concepts into low-dimensional symbols.
• Provide explicit intermediate symbols and a step-by-step execution flow for controllability and explainability.

Proposed method

• Embed a condition-flexible diffusion model within GPT to plan sequences of image-editing programs.
• Use instance normalization guidance to decouple unconditional and conditional predictions and remove dependence on hand-tuned guidance scale.
• Incorporate cross-attention to connect image features with textual prompts for spatially controllable edits.
• Define in-context visual programming with symbols like [Prompt], [RoI object], [Scenario] to enable context-free editing.
• Implement a GPT-driven Planner with operations: GPlan, PG (Prompter), Segment, Inpaint, PM (Position Manipulator).
• Execute programs through a Compiler that maps variables to values and runs operations step-by-step with explainable intermediate outputs.

Experimental results

Research questions

• RQ1 How can diffusion-based image translation be made condition-flexible without manual guidance scales?
• RQ2 Can a neuro-symbolic, visual-programming approach enable precise RoI-focused edits while maintaining global coherence?
• RQ3 Does explicit symbolic intermediate representation improve controllability and explainability of image translation?
• RQ4 Can in-context reasoning decouple high-dimensional concepts into low-dimensional symbols to support context-free editing?

Key findings

• DVP outperforms state-of-the-art baselines on fidelity and quality across diverse prompts.
• Instance normalization guidance stabilizes translation and removes sensitivity to guidance scales.
• In-context visual programming enables localized, controllable edits with explicit intermediate symbols for transparency.
• Prompter-generated annotations improve label efficiency and final image quality.
• DVP demonstrates strong RoI-focused translation while preserving background context.
• User studies and CLIP/DINO metrics show higher fidelity and quality than competitors.