[Paper Review] Mora: Enabling Generalist Video Generation via A Multi-Agent Framework

Zhengqing Yuan, Liu, Yixin+11 more|arXiv (Cornell University)|Mar 20, 2024

Advanced Vision and Imaging4 citations

TL;DR

Mora proposes a multi-agent framework that enables generalist video generation by decomposing tasks into specialized agents for prompt enhancement, image generation, video synthesis, and video editing. It achieves Sora-like capabilities across diverse tasks such as text-to-video, image-to-video, video extension, and digital world simulation, demonstrating strong performance despite a gap in quality and length compared to Sora.

ABSTRACT

Text-to-video generation has made significant strides, but replicating the capabilities of advanced systems like OpenAI Sora remains challenging due to their closed-source nature. Existing open-source methods struggle to achieve comparable performance, often hindered by ineffective agent collaboration and inadequate training data quality. In this paper, we introduce Mora, a novel multi-agent framework that leverages existing open-source modules to replicate Sora functionalities. We address these fundamental limitations by proposing three key techniques: (1) multi-agent fine-tuning with a self-modulation factor to enhance inter-agent coordination, (2) a data-free training strategy that uses large models to synthesize training data, and (3) a human-in-the-loop mechanism combined with multimodal large language models for data filtering to ensure high-quality training datasets. Our comprehensive experiments on six video generation tasks demonstrate that Mora achieves performance comparable to Sora on VBench, outperforming existing open-source methods across various tasks. Specifically, in the text-to-video generation task, Mora achieved a Video Quality score of 0.800, surpassing Sora 0.797 and outperforming all other baseline models across six key metrics. Additionally, in the image-to-video generation task, Mora achieved a perfect Dynamic Degree score of 1.00, demonstrating exceptional capability in enhancing motion realism and achieving higher Imaging Quality than Sora. These results highlight the potential of collaborative multi-agent systems and human-in-the-loop mechanisms in advancing text-to-video generation. Our code is available at \url{https://github.com/lichao-sun/Mora}.

Motivation & Objective

• Address the lack of open-source, generalist video generation models comparable to OpenAI's Sora, which remains closed-source and inaccessible to most researchers.
• Overcome limitations in existing video models that struggle with long-duration, high-resolution, and temporally consistent video generation beyond 10 seconds.
• Enable a broad spectrum of video generation tasks—including text-to-video, image-to-video, video extension, editing, connection, and digital world simulation—using a collaborative agent-based architecture.
• Replicate Sora’s generalist capabilities through modular, flexible agent pipelines that preserve visual quality and instruction adherence.
• Provide a transparent, open, and extensible framework to advance research and innovation in video generation beyond proprietary models.

Proposed method

• Decompose video generation into five core subtasks: prompt enhancement, text-to-image generation, image refinement using conditional text, image-to-video generation, and video connection.
• Deploy specialized agents for each subtask, with agents dynamically orchestrating the pipeline through looped and permuted task execution.
• Utilize pre-trained vision-language models (e.g., text-to-image diffusion models) as foundational components for high-fidelity image and video generation.
• Integrate multi-agent coordination via a meta-programming approach that enables adaptive routing and feedback between agents during inference.
• Apply iterative refinement and conditioning to maintain temporal consistency and visual coherence across generated videos.
• Leverage a modular pipeline architecture that allows plug-and-play integration of new agents or models for extended functionality.

Experimental results

Research questions

• RQ1 Can a multi-agent framework replicate the generalist video generation capabilities of Sora using open-source components?
• RQ2 How well can a collaborative agent system perform across diverse video generation tasks such as text-to-video, image-to-video, video extension, and editing?
• RQ3 What is the performance gap between an open-source multi-agent system and a closed-source SOTA model like Sora in terms of video quality, length, and instruction adherence?
• RQ4 To what extent can intermediate visual outputs (e.g., images) improve the fidelity and consistency of end-to-end video generation?
• RQ5 How do human preferences align with the outputs of such a framework, and what limitations exist in physical realism and motion control?

Key findings

• Mora successfully generates high-resolution (1024×576), temporally consistent videos of up to 12 seconds (75 frames) from text prompts, achieving visual quality comparable to Sora in specific tasks.
• The framework supports six generalist video generation tasks: text-to-video, text-conditional image-to-video, video extension, video-to-video editing, video connection, and digital world simulation.
• Mora demonstrates strong performance in prompt following and visual diversity, preserving style and content from input conditions, though it struggles with precise motion dynamics like speed and direction.
• Despite its capabilities, Mora exhibits a notable performance gap compared to Sora, particularly in generating videos longer than 12 seconds and maintaining high fidelity beyond that duration.
• Human preference evaluations reveal inconsistencies, such as illogical transitions (e.g., male to female figure disintegration), indicating a need for better physical realism and alignment with human visual expectations.
• The multi-agent approach enables flexible, modular, and extensible video generation pipelines, offering a viable open-source alternative to closed-source models like Sora.