[Paper Review] FreeNoise: Tuning-Free Longer Video Diffusion via Noise Rescheduling

Haonan Qiu, Menghan Xia+5 more|arXiv (Cornell University)|Oct 23, 2023

Generative Adversarial Networks and Image Synthesis4 citations

TL;DR

FreeNoise proposes a tuning-free, computationally efficient method to generate high-fidelity long videos from pretrained diffusion models by rescheduling initial noise for long-range temporal correlation and enabling window-based attention fusion. It further introduces motion injection to support multi-text conditioning without fine-tuning, achieving state-of-the-art video quality with only 17% additional inference time—significantly lower than the 255% cost of prior methods.

ABSTRACT

With the availability of large-scale video datasets and the advances of diffusion models, text-driven video generation has achieved substantial progress. However, existing video generation models are typically trained on a limited number of frames, resulting in the inability to generate high-fidelity long videos during inference. Furthermore, these models only support single-text conditions, whereas real-life scenarios often require multi-text conditions as the video content changes over time. To tackle these challenges, this study explores the potential of extending the text-driven capability to generate longer videos conditioned on multiple texts. 1) We first analyze the impact of initial noise in video diffusion models. Then building upon the observation of noise, we propose FreeNoise, a tuning-free and time-efficient paradigm to enhance the generative capabilities of pretrained video diffusion models while preserving content consistency. Specifically, instead of initializing noises for all frames, we reschedule a sequence of noises for long-range correlation and perform temporal attention over them by window-based function. 2) Additionally, we design a novel motion injection method to support the generation of videos conditioned on multiple text prompts. Extensive experiments validate the superiority of our paradigm in extending the generative capabilities of video diffusion models. It is noteworthy that compared with the previous best-performing method which brought about 255% extra time cost, our method incurs only negligible time cost of approximately 17%. Generated video samples are available at our website: http://haonanqiu.com/projects/FreeNoise.html.

Motivation & Objective

• To address the limitation of existing video diffusion models that are trained on short video clips and fail to generate high-fidelity long videos during inference.
• To enable multi-text conditioned video generation without fine-tuning, supporting dynamic content changes over time.
• To develop a method that enhances long video generation while preserving content consistency and minimizing computational overhead.
• To investigate the role of initial noise in temporal modeling and its impact on video generation quality.

Proposed method

• Proposes FreeNoise, a tuning-free paradigm that reschedules a sequence of fixed random noise frames to establish long-range temporal correlation for longer video generation.
• Introduces Local Noise Shuffling to generate noise sequences with internal randomness and long-range coherence, enabling consistent content across extended video lengths.
• Employs Window-Based Attention Fusion to allow pre-trained temporal attention modules to process longer sequences by applying overlapped temporal windowing, with no additional computation cost to other model components.
• Designs Motion Injection to support multi-prompt video generation by injecting new motion cues at specific denoising steps, aligned with object shape recovery stages.
• Uses a two-stage denoising strategy where the first prompt controls layout and object shapes, and the second prompt is injected at later steps to introduce motion changes.
• Applies the method to multiple pretrained models (VideoCrafter, AnimateDiff, LaVie) without any fine-tuning, ensuring broad applicability and efficiency.

Experimental results

Research questions

• RQ1 How does the initial noise distribution affect the quality and consistency of long video generation in diffusion models?
• RQ2 Can noise rescheduling enable longer video generation without fine-tuning or significant computational cost?
• RQ3 How can multi-text conditions be effectively integrated into video diffusion models to support dynamic scene transitions?
• RQ4 What is the optimal timing and layer for injecting new motion cues to preserve visual consistency and enable smooth transitions?
• RQ5 Can a window-based attention mechanism extend the receptive field of pretrained temporal modules without introducing additional inference overhead?

Key findings

• FreeNoise achieves state-of-the-art video quality in long video generation while incurring only 17% additional inference time, compared to 255% for the previous best method.
• The ablation study confirms that a stride of 4 in noise rescheduling provides sufficient content consistency without doubling inference cost.
• Motion Injection successfully enables smooth transitions between actions (e.g., from running to standing) in a single video, with minimal visual artifacts.
• The method preserves the appearance of main subjects (e.g., horses, bowls) across long video sequences, even when new motion is injected.
• Window-based attention fusion enables the model to process arbitrarily long video lengths using only the original pretrained temporal attention modules.
• The approach maintains content consistency across long-range frames, as validated by qualitative and quantitative results on multiple benchmarks.