[論文レビュー] VDT: General-purpose Video Diffusion Transformers via Mask Modeling

Haoyu Lu, Guoxing Yang+5 more|arXiv (Cornell University)|May 22, 2023

Generative Adversarial Networks and Image Synthesis被引用数 9

ひとこと要約

Video Diffusion Transformer (VDT) を紹介します。時空の注意機構と統一的な時空マスクモデリング機構を用いた、 unconditional generation、prediction、interpolation、animation、および completion を扱うための、動画生成のための純粋なトランスフォーマーベースの拡散モデルです。

ABSTRACT

This work introduces Video Diffusion Transformer (VDT), which pioneers the use of transformers in diffusion-based video generation. It features transformer blocks with modularized temporal and spatial attention modules to leverage the rich spatial-temporal representation inherited in transformers. We also propose a unified spatial-temporal mask modeling mechanism, seamlessly integrated with the model, to cater to diverse video generation scenarios. VDT offers several appealing benefits. 1) It excels at capturing temporal dependencies to produce temporally consistent video frames and even simulate the physics and dynamics of 3D objects over time. 2) It facilitates flexible conditioning information, \eg, simple concatenation in the token space, effectively unifying different token lengths and modalities. 3) Pairing with our proposed spatial-temporal mask modeling mechanism, it becomes a general-purpose video diffuser for harnessing a range of tasks, including unconditional generation, video prediction, interpolation, animation, and completion, etc. Extensive experiments on these tasks spanning various scenarios, including autonomous driving, natural weather, human action, and physics-based simulation, demonstrate the effectiveness of VDT. Additionally, we present comprehensive studies on how \model handles conditioning information with the mask modeling mechanism, which we believe will benefit future research and advance the field. Project page: https:VDT-2023.github.io

研究の動機と目的

• Pioneer the use of transformers as backbones in diffusion-based video generation.
• Develop a unified spatial-temporal mask modeling mechanism to support diverse video tasks.
• Enable flexible conditioning and efficient handling of varying input lengths/modalities in video generation.
• Demonstrate strong performance across unconditional generation, video prediction, interpolation, and completion on multiple datasets.

提案手法

• Use a pure transformer architecture with temporal and spatial attention within each VDT block.
• Project videos into a latent space via a pre-trained VAE tokenizer to reduce computation.
• Incorporate temporal position embeddings and spatial position embeddings to learn spatiotemporal information.
• Apply adaptive group normalization to inject diffusion time information into transformer blocks.
• Explore conditioning schemes for video prediction, including adaptive layer normalization, cross-attention, and token concatenation.
• Introduce a unified spatial-temporal mask modeling mechanism to blend conditional frames and noise for multiple tasks.

実験結果

リサーチクエスチョン

• RQ1 Can a transformer-based diffusion model capture temporal dependencies effectively for high-quality, temporally consistent video generation?
• RQ2 How can conditioning information be incorporated and unified across diverse video generation tasks (unconditional generation, prediction, interpolation, animation, completion)?
• RQ3 Does a unified spatial-temporal mask modeling approach enable a single model to perform multiple video generation tasks without architectural changes?
• RQ4 Which conditioning strategy yields the best convergence speed and sample quality for video prediction?
• RQ5 How does VDT compare to state-of-the-art diffusion and other generative methods on standard video benchmarks?

主な発見

• VDT は unconditional video generation（例：UCF-101 で）において最先端の手法と競合するかそれ以上の性能を達成し、複数の GAN および拡散ベースのベースラインを上回る。
• Video prediction の conditioning 戦略としての token concatenation は、探索したスキームの中で最も速い収束と最良のサンプル品質（FVD/SSIM）をもたらす。
• 統一的な時空マスクモデリングにより、VDT は条件付きフレームとノイズを単一のフレームワーク内で統合し、無条件生成、双方向予測、任意の補間、画像から動画生成、および時空的完成といった多様なタスクを処理できる。
• VDT は強力な時系列モデリングを示し、Cityscapes および Physion データセットでの動画予測においてベースラインと同等以上を達成し、色の整合性と運動の一貫性を維持する。
• Physion では、VDT は物理衝突予測のための VQA 精度がシーン中心の方法より高く（65.3% 対 63.1% まで）、堅牢な物理動画予測能力を示す。
• 画像事前学習後の結合訓練により、エンドツーエンドの時空的訓練よりも効率と性能が向上する。