[Paper Review] Structure and Content-Guided Video Synthesis with Diffusion Models

Patrick Esser, Johnathan Chiu+3 more|arXiv (Cornell University)|Feb 6, 2023

Generative Adversarial Networks and Image Synthesis17 citations

TL;DR

The paper presents a structure and content-guided latent video diffusion model that edits videos according to text or image prompts while preserving input structure, using joint image-video training, depth-based structure, and a novel guidance method to control temporal consistency.

ABSTRACT

Text-guided generative diffusion models unlock powerful image creation and editing tools. While these have been extended to video generation, current approaches that edit the content of existing footage while retaining structure require expensive re-training for every input or rely on error-prone propagation of image edits across frames. In this work, we present a structure and content-guided video diffusion model that edits videos based on visual or textual descriptions of the desired output. Conflicts between user-provided content edits and structure representations occur due to insufficient disentanglement between the two aspects. As a solution, we show that training on monocular depth estimates with varying levels of detail provides control over structure and content fidelity. Our model is trained jointly on images and videos which also exposes explicit control of temporal consistency through a novel guidance method. Our experiments demonstrate a wide variety of successes; fine-grained control over output characteristics, customization based on a few reference images, and a strong user preference towards results by our model.

Motivation & Objective

• Develop a controllable video diffusion model that edits content while preserving structure.
• Enable text- and image-guided video edits without per-video training.
• Achieve explicit control over temporal, content, and structure fidelity.
• Explore training on depth-based structure representations with varying detail to modulate fidelity.
• Demonstrate customization and user preference for edits.

Proposed method

• Extend latent diffusion models to the spatio-temporal domain by adding temporal layers to a pre-trained image model.
• Represent structure with monocular depth estimates and content with CLIP-based embeddings.
• Train jointly on images and videos to enable inference-time temporal control via a temporal guidance scale.
• Condition the model on structure s (via concatenation) and content c (via cross-attention) during denoising.
• Use depth maps with varying blurring t_s to control structure fidelity during training and inference.
• Apply classifier-free diffusion guidance with content and temporal guidance scales to modulate prompt fidelity and temporal consistency.

Experimental results

Research questions

• RQ1 How can diffusion models edit video content while preserving the original structure of the input video?
• RQ2 Can joint training on images and videos provide explicit temporal consistency control at inference time?
• RQ3 How can depth-based structure representations and CLIP-based content representations be effectively conditioned in a video diffusion model?
• RQ4 To what extent can editing fidelity and temporal smoothness be controlled via sampling guidance and structure detail levels?

Key findings

• The model enables fine-grained control over temporal consistency, structure fidelity, and content edits at inference time.
• Joint training on image and video data improves temporal consistency compared to image-only approaches.
• Depth-based structure representations with varying detail (t_s) allow control over how much structure is preserved in edits.
• Content can be steered by text prompts or example images via CLIP embeddings and a learned prior to convert text to image embeddings.
• A novel temporal guidance mechanism (ω_t) during sampling improves frame-to-frame coherence while maintaining prompt adherence.
• User studies show the approach is preferred over several baselines for text- and image-guided video editing.