[Paper Review] WorldGPT: A Sora-Inspired Video AI Agent as Rich World Models from Text and Image Inputs

Deshun Yang, Luhui Hu+6 more|arXiv (Cornell University)|Mar 10, 2024

Multimodal Machine Learning Applications4 citations

TL;DR

WorldGPT proposes a Sora-inspired video AI agent that generates high-quality, temporally consistent videos from text and image inputs using a two-stage pipeline: a GPT-powered prompt enhancer for precise instruction refinement and a diffusion-based video generation module leveraging key frames for improved motion smoothness. The method achieves state-of-the-art performance in text-video alignment, temporal consistency, and motion quality, outperforming DynamiCrafter and I2VGen-XL in human evaluations.

ABSTRACT

Several text-to-video diffusion models have demonstrated commendable capabilities in synthesizing high-quality video content. However, it remains a formidable challenge pertaining to maintaining temporal consistency and ensuring action smoothness throughout the generated sequences. In this paper, we present an innovative video generation AI agent that harnesses the power of Sora-inspired multimodal learning to build skilled world models framework based on textual prompts and accompanying images. The framework includes two parts: prompt enhancer and full video translation. The first part employs the capabilities of ChatGPT to meticulously distill and proactively construct precise prompts for each subsequent step, thereby guaranteeing the utmost accuracy in prompt communication and accurate execution in following model operations. The second part employ compatible with existing advanced diffusion techniques to expansively generate and refine the key frame at the conclusion of a video. Then we can expertly harness the power of leading and trailing key frames to craft videos with enhanced temporal consistency and action smoothness. The experimental results confirm that our method has strong effectiveness and novelty in constructing world models from text and image inputs over the other methods.

Motivation & Objective

• To address the challenge of maintaining temporal consistency and action smoothness in text-to-video generation.
• To improve video generation quality by integrating multimodal inputs (text and images) into a unified world model framework.
• To enhance user controllability through precise, GPT-optimized prompts that guide video generation.
• To develop a pipeline that leverages existing advanced diffusion models for high-fidelity video synthesis with improved spatiotemporal coherence.
• To evaluate the method on zero-shot benchmarks and human preference studies, demonstrating superiority in alignment and motion quality.

Proposed method

• The framework employs a two-part pipeline: a prompt enhancer using GPT to refine and proactively construct precise textual instructions for video generation.
• The full video translation stage uses compatible diffusion techniques to generate and refine key frames at the start and end of video sequences.
• Temporal consistency is enhanced by leveraging leading and trailing key frames to guide the generation of intermediate frames.
• The method integrates dynamic scene modeling and multimodal fusion to align video content with both textual descriptions and input images.
• It applies a loss function minimizing the difference between generated frames and ground truth, using optical flow (F) and appearance change fields (B) to ensure spatiotemporal coherence.
• The system is evaluated using AIGCBench benchmarks and human preference studies, comparing against DynamiCrafter and I2VGen-XL.

Experimental results

Research questions

• RQ1 Can a GPT-enhanced prompt generation strategy significantly improve text-video alignment in video generation?
• RQ2 How effectively can key frame-based diffusion modeling enhance temporal consistency and motion smoothness in generated videos?
• RQ3 To what extent does multimodal fusion of text and image inputs improve the controllability and diversity of video generation?
• RQ4 How does the proposed method compare to SOTA models like DynamiCrafter and I2VGen-XL in zero-shot video generation benchmarks?
• RQ5 What is the human preference for video quality, motion realism, and alignment in user evaluations?

Key findings

• WorldGPT achieved a GenVideo-Text Clip score of 0.307 on AIGCBench, outperforming DynamiCrafter (0.24) and I2VGen-XL (0.26) in text-video alignment.
• The method achieved a temporal consistency score of 0.992 on the GenVideo Clip metric, surpassing DynamiCrafter (0.980) and I2VGen-XL (0.960).
• In human evaluation, 66.8% of participants preferred WorldGPT’s motion quality over DynamiCrafter, and 68.7% preferred its text-video alignment over I2VGen-XL.
• WorldGPT demonstrated superior performance in handling complex, out-of-distribution textual prompts, generating videos that better reflect novel scene descriptions not present in the input image.
• The model achieved a DOVER score of 0.521, indicating strong frame-level quality, and a GenVideo-RefVideo SSIM of 0.374, showing improved alignment with reference videos.
• Qualitative results confirmed that WorldGPT better captures semantic distinctions between prompts, especially when prompts diverge significantly from the input image, unlike DynamiCrafter, which often produces near-identical outputs.