[Paper Review] Dreamix: Video Diffusion Models are General Video Editors

Eyal Molad, Eliahu Horwitz+6 more|arXiv (Cornell University)|Feb 2, 2023

Generative Adversarial Networks and Image Synthesis43 citations

TL;DR

Dreamix introduces a text-guided video editor based on diffusion models that preserves fidelity to the original video while enabling appearance and motion edits, via mixed finetuning on video frames and degraded input videos. It also enables image animation and subject-driven video generation.

ABSTRACT

Text-driven image and video diffusion models have recently achieved unprecedented generation realism. While diffusion models have been successfully applied for image editing, very few works have done so for video editing. We present the first diffusion-based method that is able to perform text-based motion and appearance editing of general videos. Our approach uses a video diffusion model to combine, at inference time, the low-resolution spatio-temporal information from the original video with new, high resolution information that it synthesized to align with the guiding text prompt. As obtaining high-fidelity to the original video requires retaining some of its high-resolution information, we add a preliminary stage of finetuning the model on the original video, significantly boosting fidelity. We propose to improve motion editability by a new, mixed objective that jointly finetunes with full temporal attention and with temporal attention masking. We further introduce a new framework for image animation. We first transform the image into a coarse video by simple image processing operations such as replication and perspective geometric projections, and then use our general video editor to animate it. As a further application, we can use our method for subject-driven video generation. Extensive qualitative and numerical experiments showcase the remarkable editing ability of our method and establish its superior performance compared to baseline methods.

Motivation & Objective

• Enable general text-based appearance and motion editing of real-world videos using a video diffusion model.
• Improve motion editability through a mixed finetuning strategy that combines full temporal attention with temporal attention masking.
• Propose a framework for image-to-video animation by converting images into coarse videos and editing them with the Dreamix editor.
• Demonstrate subject-driven video generation by fine-tuning on a set of subject images to enable guided video creation.

Proposed method

• Use a cascaded video diffusion model (Imagen-Video base) to generate high-resolution edited videos conditioned on text prompts and a degraded input video.
• Corrupt the input video by downsampling and adding noise, then reverse the diffusion process conditioned on the prompt to achieve text-guided editing (inversion of corruptions).
• Introduce mixed finetuning that jointly optimizes on the input video (appearance and motion) and on an unordered set of frames with masked temporal attention to preserve frame-wise details while enabling motion edits.
• Train with two objectives: L_vid to reconstruct the input video given noisy degraded input, and L_frame to reconstruct individual frames with masked temporal attention (D^a).
• Provide an image-to-video workflow by creating a coarse video from an image via replication or perspective transforms, then applying Dreamix editing to obtain a high-quality edited video.
• Enable subject-driven video generation by finetuning on a set of subject images with a special token t* and using a text prompt for generation.

Experimental results

Research questions

• RQ1 Can a text-guided video diffusion model edit both appearance and motion while maintaining fidelity to the original video?
• RQ2 Does mixed finetuning improve motion Editability and overall edit quality compared to video-only or no finetuning?
• RQ3 Can the Dreamix framework be extended to image animation and subject-driven video generation?
• RQ4 What are the trade-offs and requirements (e.g., hyperparameters, computational cost) for high-quality video editing with diffusion models?

Key findings

• Dreamix enables general text-based appearance and motion editing of real-world videos with improved temporal consistency.
• Mixed finetuning with masked temporal attention significantly improves motion edits and preserves background details compared to non-mixed or video-only finetuning.
• Dreamix provides a framework for image animation by converting images to coarse videos and editing them with the Dreamix editor.
• Subject-driven video generation is achievable by finetuning on a set of subject images and generating videos conditioned on text prompts.
• Compared to baselines, Dreamix achieves higher user-rated quality, fidelity, and prompt alignment, with a 73% success rate vs 40% (ImgenVid) and 15% (PnP).
• Quantitative baselines show Dreamix outperforms frame-by-frame editing approaches in temporal consistency and fidelity.