[Paper Review] Real-Time Video Generation with Pyramid Attention Broadcast

Xinrong Zhao, Xiaolong Jin+2 more|arXiv (Cornell University)|Aug 22, 2024

Advanced Vision and Imaging4 citations

TL;DR

This paper proposes Pyramid Attention Broadcast (PAB), a training-free, real-time video generation method for DiT-based models that reduces attention redundancy by broadcasting attention outputs across diffusion steps in a pyramid-style strategy based on variance. PAB achieves up to 20.6 FPS for 720p video generation with 10.5× speedup and zero quality degradation across Open-Sora, Open-Sora-Plan, and Latte models.

ABSTRACT

We present Pyramid Attention Broadcast (PAB), a real-time, high quality and training-free approach for DiT-based video generation. Our method is founded on the observation that attention difference in the diffusion process exhibits a U-shaped pattern, indicating significant redundancy. We mitigate this by broadcasting attention outputs to subsequent steps in a pyramid style. It applies different broadcast strategies to each attention based on their variance for best efficiency. We further introduce broadcast sequence parallel for more efficient distributed inference. PAB demonstrates up to 10.5x speedup across three models compared to baselines, achieving real-time generation for up to 720p videos. We anticipate that our simple yet effective method will serve as a robust baseline and facilitate future research and application for video generation.

Motivation & Objective

• Address the high inference latency of DiT-based video diffusion models, which limits practical deployment despite high-quality outputs.
• Overcome the inefficiency of attention computation in diffusion transformers, especially in video generation with complex spatiotemporal dependencies.
• Develop a training-free acceleration method that avoids retraining while maintaining generation quality across diverse DiT-based video models.
• Enable real-time inference (≥20 FPS) for high-resolution (up to 720p) video generation without architectural or distributional changes.
• Design a scalable, generalized acceleration framework applicable to future DiT-based video models without retraining.

Proposed method

• Propose Pyramid Attention Broadcast (PAB), a training-free method that reuses attention outputs across diffusion steps by exploiting a U-shaped pattern of attention differences.
• Apply different broadcast ranges to spatial, temporal, and cross-attention heads based on their variance—spatial attention has the widest broadcast, cross-attention the narrowest—forming a pyramid strategy.
• Introduce broadcast sequence parallelism to reduce communication costs in distributed inference, enabling efficient multi-GPU deployment.
• Use mean squared error (MSE) of attention outputs between consecutive steps to quantify and guide broadcast strategy selection.
• Extend the broadcast mechanism beyond attention layers to include MLP layers and diffusion steps, improving overall efficiency.
• Integrate PAB into existing DiT-based video models (e.g., Open-Sora, Latte) without modifying the model architecture or requiring retraining.

Experimental results

Research questions

• RQ1 Can attention redundancy in DiT-based video diffusion models be effectively exploited to accelerate inference without quality loss?
• RQ2 Does the U-shaped pattern of attention differences across diffusion steps enable a structured, variance-based broadcast strategy for optimal efficiency?
• RQ3 Can a pyramid-style broadcast strategy, tailored to attention type (spatial, temporal, cross), outperform uniform or fixed-range broadcast methods?
• RQ4 How effective is broadcast sequence parallelism in reducing communication overhead during distributed inference for video generation?
• RQ5 Can PAB achieve real-time inference (≥20 FPS) for high-resolution (720p) video generation across multiple state-of-the-art DiT-based models?

Key findings

• PAB achieves up to 10.5× speedup in inference time across three leading DiT-based video models: Open-Sora, Open-Sora-Plan, and Latte.
• The method enables real-time video generation at 20.6 FPS for 720p videos, with latency measured on H100 GPUs.
• The U-shaped pattern of attention differences is empirically validated, with minimal changes in the middle 70% of diffusion steps.
• Broadcasting based on attention variance improves efficiency: spatial attention (most variable) uses wider ranges, while cross-attention (most stable) uses narrower ranges.
• Broadcast sequence parallelism reduces communication costs in distributed inference, enabling scalable and efficient multi-GPU deployment.
• PAB maintains original generation quality without any fine-tuning or retraining, demonstrating its generalization and robustness across diverse models.