[Paper Review] Make-A-Protagonist: Generic Video Editing with An Ensemble of Experts

Yuyang Zhao, Enze Xie+3 more|arXiv (Cornell University)|May 15, 2023

Generative Adversarial Networks and Image Synthesis4 citations

TL;DR

Make-A-Protagonist introduces a generic video editing framework that enables fine-grained, personalized video editing using both textual and visual clues, leveraging an ensemble of pre-trained, frozen experts for zero-shot parsing of protagonists. It achieves high-fidelity editing by combining mask-guided denoising sampling with a diffusion-based video generation model, outperforming text-only or single-reference methods in protagonist substitution, background editing, and text-to-video editing tasks.

ABSTRACT

The text-driven image and video diffusion models have achieved unprecedented success in generating realistic and diverse content. Recently, the editing and variation of existing images and videos in diffusion-based generative models have garnered significant attention. However, previous works are limited to editing content with text or providing coarse personalization using a single visual clue, rendering them unsuitable for indescribable content that requires fine-grained and detailed control. In this regard, we propose a generic video editing framework called Make-A-Protagonist, which utilizes textual and visual clues to edit videos with the goal of empowering individuals to become the protagonists. Specifically, we leverage multiple experts to parse source video, target visual and textual clues, and propose a visual-textual-based video generation model that employs mask-guided denoising sampling to generate the desired output. Extensive results demonstrate the versatile and remarkable editing capabilities of Make-A-Protagonist.

Motivation & Objective

• To address the limitation of existing text-only or single-reference video editing models in handling indescribable or personalized content.
• To enable fine-grained, controllable video editing where users can replace protagonists with personal identities not easily described in text.
• To eliminate the need for fine-tuning or data annotation by leveraging pre-trained, frozen models as experts.
• To achieve consistent and high-quality video generation while preserving background structure and motion fidelity.

Proposed method

• Utilizes BLIP-2 for video captioning and video question answering to extract textual descriptions of the protagonist.
• Employs Grounded-SAM and XMem to generate instance masks from textual descriptions, enabling visual protagonist localization and tracking across frames.
• Encodes visual and textual clues using CLIP Vision and DALL-E 2 Prior to generate image and text embeddings.
• Integrates ControlNet with depth and pose conditions to maintain structural consistency during video generation.
• Employs a mask-guided denoising sampling strategy to fuse visual and textual embeddings with segmentation masks during diffusion sampling.
• Leverages a pre-trained image-to-video diffusion model with temporal modules, initialized from a text-to-image prior, to generate edited videos.

Experimental results

Research questions

• RQ1 Can a generic video editing framework effectively replace protagonists in videos using both visual and textual clues without requiring fine-tuning?
• RQ2 How does the ensemble of pre-trained experts compare to fine-tuned models in handling personalized, non-identifiable protagonists?
• RQ3 To what extent can mask-guided denoising sampling improve editing fidelity and consistency compared to standard text-only editing?
• RQ4 How does the framework perform in editing tasks beyond protagonist substitution, such as background editing and text-to-video editing with protagonists?
• RQ5 What are the main failure modes and biases introduced by pre-trained models in such editing scenarios?

Key findings

• Make-A-Protagonist successfully enables editing of protagonists in videos using both visual and textual clues, even when the protagonist is an ordinary individual not well-represented in text embeddings.
• The framework achieves high-quality editing results without fine-tuning, demonstrating strong zero-shot generalization across diverse editing tasks.
• Qualitative comparisons show that DreamBooth-V overfits to reference gestures, producing unnatural motion, while Make-A-Protagonist maintains better motion realism despite lower DINO scores.
• Failure cases reveal overfitting to the source video and bias in pre-trained text-to-image models, particularly when prompts like 'tiger' are ambiguous or misaligned with model prior.
• Using more precise prompts such as 'white tiger' significantly improves subject fidelity and reduces intermediate or distorted subject generation.