[Paper Review] MagicProp: Diffusion-based Video Editing via Motion-aware Appearance Propagation

Hanshu Yan, Jun Hao Liew+3 more|arXiv (Cornell University)|Sep 2, 2023

Generative Adversarial Networks and Image SynthesisComputer Science3 citations

TL;DR

MagicProp is a diffusion-based video editing framework that decouples editing into appearance editing and motion-aware appearance propagation. It uses a reference frame edited via image diffusion models and autoregressively generates subsequent frames using a conditional diffusion model (PropDPM) that integrates motion cues and reference appearance, achieving high temporal consistency and flexible regional editing with minimal artifacts.

ABSTRACT

This paper addresses the issue of modifying the visual appearance of videos while preserving their motion. A novel framework, named MagicProp, is proposed, which disentangles the video editing process into two stages: appearance editing and motion-aware appearance propagation. In the first stage, MagicProp selects a single frame from the input video and applies image-editing techniques to modify the content and/or style of the frame. The flexibility of these techniques enables the editing of arbitrary regions within the frame. In the second stage, MagicProp employs the edited frame as an appearance reference and generates the remaining frames using an autoregressive rendering approach. To achieve this, a diffusion-based conditional generation model, called PropDPM, is developed, which synthesizes the target frame by conditioning on the reference appearance, the target motion, and its previous appearance. The autoregressive editing approach ensures temporal consistency in the resulting videos. Overall, MagicProp combines the flexibility of image-editing techniques with the superior temporal consistency of autoregressive modeling, enabling flexible editing of object types and aesthetic styles in arbitrary regions of input videos while maintaining good temporal consistency across frames. Extensive experiments in various video editing scenarios demonstrate the effectiveness of MagicProp.

Motivation & Objective

• Address the challenge of maintaining temporal consistency while enabling flexible, region-specific video editing.
• Decouple video editing into two stages: appearance editing and motion-aware propagation to balance flexibility and consistency.
• Enable editing of arbitrary regions (foreground, background, or global) using powerful image editing techniques.
• Develop a conditional diffusion model that preserves appearance fidelity and motion coherence across frames.
• Overcome limitations of existing methods that sacrifice editability for consistency or vice versa.

Proposed method

• Select a single reference frame from the input video and apply image-editing techniques (e.g., text prompts, segmentation masks) to modify its content or style.
• Use an autoregressive diffusion model, PropDPM, to generate remaining frames conditioned on the reference frame, motion sequence (e.g., depth maps), and previous frame.
• Design a lightweight appearance adaptor to inject both class-level (CLIP) and pixel-level (VAE) features from the reference frame into the diffusion model’s cross-attention layers.
• Apply the zero-terminal signal-to-noise ratio (SNR) training strategy to align noise schedules between training and inference, reducing color-shifting and improving visual quality.
• Train PropDPM on a combination of WebVid-10M and a private high-resolution video dataset, using triplets of frames (reference, previous, target) for autoregressive learning.
• Utilize control signals such as depth maps and segmentation masks to guide precise spatial editing while preserving motion consistency.

Figure 1: Video editing via MagicProp: global, background, and foreground editing are all supported.

Experimental results

Research questions

• RQ1 Can a two-stage framework combining image editing and autoregressive diffusion achieve both high editability and strong temporal consistency in video editing?
• RQ2 How effectively can a diffusion model conditioned on reference appearance and motion cues preserve visual fidelity across frames?
• RQ3 To what extent does the zero-terminal SNR training schedule improve color and style alignment between generated frames and the reference?
• RQ4 How well does the proposed appearance adaptor integrate semantic and pixel-level features to guide accurate appearance transfer?
• RQ5 What is the upper limit of video length that can be edited with acceptable quality using an autoregressive diffusion approach?

Key findings

• MagicProp achieves high-quality video editing with strong temporal consistency, even for complex edits involving multiple objects or style transfers.
• The zero-terminal SNR training strategy significantly reduces color-shifting artifacts, improving visual fidelity compared to standard DDPM noise schedules.
• The appearance adaptor effectively fuses CLIP-based semantic tokens and VAE-based pixel features, enabling accurate appearance transfer to generated frames.
• The framework supports flexible editing of arbitrary regions—foreground, background, or global—using text prompts or segmentation masks.
• The method maintains high visual quality for videos up to 30 frames; beyond this, degradation such as over-smoothing and artifacts emerge due to autoregressive error accumulation.
• Extensive experiments demonstrate that MagicProp outperforms existing image-based and sequence-based video editing methods in both editability and temporal coherence.

Figure 2: The pipeline of MagicProp.