[Paper Review] CVPR 2023 Text Guided Video Editing Competition

Jay Zhangjie Wu, Xiuyu Li+18 more|arXiv (Cornell University)|Oct 24, 2023

Generative Adversarial Networks and Image Synthesis6 citations

TL;DR

This paper introduces the CVPR 2023 Text Guided Video Editing (TGVE) competition, proposing a standardized benchmark dataset of 76 videos with 4 edit prompts each for style, background, object, and multi-edit changes. The winning method, Team CAMP's Two-Stage Video Editing (2SVE) pipeline, leverages ControlNet, Segment Anything Model (SAM), OpenCLIP, and fine-tuned MSVD data to achieve superior editing fidelity and temporal consistency, outperforming the baseline in 59.1% of human evaluations.

ABSTRACT

Humans watch more than a billion hours of video per day. Most of this video was edited manually, which is a tedious process. However, AI-enabled video-generation and video-editing is on the rise. Building on text-to-image models like Stable Diffusion and Imagen, generative AI has improved dramatically on video tasks. But it's hard to evaluate progress in these video tasks because there is no standard benchmark. So, we propose a new dataset for text-guided video editing (TGVE), and we run a competition at CVPR to evaluate models on our TGVE dataset. In this paper we present a retrospective on the competition and describe the winning method. The competition dataset is available at https://sites.google.com/view/loveucvpr23/track4.

Motivation & Objective

• To address the lack of a standardized benchmark for evaluating text-guided video editing (TGVE) models, which hinders progress and model comparison in the field.
• To create a publicly available, diverse, and high-quality dataset of 76 videos with 4 edit prompts per video, covering style, background, object, and multi-changes.
• To organize a competition at CVPR 2023 to evaluate TGVE models using both automated and human evaluation metrics.
• To identify and analyze the strengths and weaknesses of existing TGVE methods through a controlled, open competition with transparent evaluation.
• To establish a reproducible, open benchmark that enables future research and development in text-guided video editing.

Proposed method

• The competition dataset was constructed from 100 publicly licensed videos (DAVIS, YouTube, Videvo), filtered to 76 for quality and diversity, each cropped and downsampled to 480×480 resolution with 32 or 128 frames.
• Each video was annotated with a ground-truth caption and four edit prompts: style-change, background-change, object-change, or multiple-changes, created via iterative prompting and refinement using LLMs and human validation.
• The winning method, Two-Stage Video Editing (2SVE), first generates edit-aligned frames using a diffusion model conditioned on text and image prompts, then refines results using ControlNet and VPA (Video Prompt Anchor) for spatial and temporal consistency.
• The 2SVE pipeline incorporates Segment Anything Model (SAM) for precise object masking, OpenCLIP for improved text-image alignment, and fine-tunes on the MSVD dataset to enhance prompt-video alignment.
• A novel Multi-frame Video Rendering (MVSD) technique was introduced, which uses the first and previous frame outputs as reference for inpainting, improving style coherence and reducing flicker across frames.
• The system uses Target Segment Process (TSP) to refine inpainting masks, minimizing interference from adjacent frames while preserving structural and motion consistency.

Experimental results

Research questions

• RQ1 What are the key challenges in evaluating text-guided video editing models, and how can a standardized benchmark be designed to fairly compare different methods?
• RQ2 How do different architectural components—such as ControlNet, SAM, and CLIP—contribute to editing fidelity and temporal consistency in TGVE?
• RQ3 To what extent can a two-stage training and inference pipeline improve video editing quality compared to end-to-end methods?
• RQ4 How do human evaluators perceive the realism, consistency, and prompt alignment of generated edits across diverse editing types?
• RQ5 What role does fine-tuning on existing datasets like MSVD play in improving prompt-video alignment and edit quality?

Key findings

• The TGVE 2023 competition dataset comprises 76 videos with 304 edit prompts, covering four edit types: style, background, object, and multiple changes, and is publicly available at the competition site.
• Team CAMP's Two-Stage Video Editing (2SVE) pipeline won the competition, outperforming the baseline model in 59.1% of human evaluations across all edit types.
• The 2SVE method achieved high visual quality and strong temporal consistency, particularly in style-transfer and object-editing tasks, with minimal flicker and preserved structural details.
• The integration of SAM for instance segmentation and ControlNet for spatial control significantly improved localization accuracy and reduced unwanted edits.
• Fine-tuning on the MSVD dataset improved prompt-video alignment, especially in complex edits involving multiple changes or subtle style shifts.
• Human evaluation confirmed that the 2SVE pipeline produced more natural, coherent, and prompt-aligned edits than baseline models, particularly in preserving background and motion structure.