[Paper Review] AnimateLCM: Computation-Efficient Personalized Style Video Generation without Personalized Video Data

Fuyun Wang, Zhaoyang Huang+6 more|arXiv (Cornell University)|Feb 1, 2024

Human Motion and Animation4 citations

TL;DR

AnimateLCM accelerates high-fidelity personalized video generation using a decoupled consistency learning strategy that separately distills image and motion priors, enabling video generation in just 4 inference steps. It achieves state-of-the-art results with minimal steps, supports plug-and-play adapters without speed loss, and is compatible with personalized image diffusion weights for diverse styles.

ABSTRACT

This paper introduces an effective method for computation-efficient personalized style video generation without requiring access to any personalized video data. It reduces the necessary generation time of similarly sized video diffusion models from 25 seconds to around 1 second while maintaining the same level of performance. The method's effectiveness lies in its dual-level decoupling learning approach: 1) separating the learning of video style from video generation acceleration, which allows for personalized style video generation without any personalized style video data, and 2) separating the acceleration of image generation from the acceleration of video motion generation, enhancing training efficiency and mitigating the negative effects of low-quality video data.

Motivation & Objective

• To accelerate video generation in diffusion models while maintaining high visual quality.
• To address the high computational cost of iterative denoising in video diffusion by enabling inference in minimal steps.
• To enable compatibility with personalized image diffusion models and community adapters without sacrificing speed or quality.
• To develop a training strategy that improves efficiency and generation quality through decoupled distillation of image and motion priors.

Proposed method

• Proposes a decoupled consistency learning strategy that first distills image generation priors from high-quality image datasets before distilling motion priors from video data.
• Applies 3D inflation to both image and image consistency models to extend them to video generation, with a specially designed initialization to reduce feature corruption.
• Trains a video consistency model in the latent space by predicting solutions to classifier-free guidance-augmented probability flow ODEs.
• Introduces a teacher-free adapter adaptation strategy that enables direct integration of existing adapters (e.g., ControlNet, T2I-Adapter) or training from scratch without requiring a teacher model.
• Leverages pre-trained stable diffusion weights and allows seamless replacement of spatial weights with personalized image diffusion models for style transfer.

Experimental results

Research questions

• RQ1 Can consistency learning be effectively decoupled between image and motion priors to improve training efficiency and generation quality in video diffusion?
• RQ2 How can existing community adapters be adapted to a distilled video consistency model without degrading inference speed?
• RQ3 Can personalized image diffusion weights be directly used in a video consistency model to generate high-fidelity, style-consistent videos?
• RQ4 Does the proposed method achieve superior performance in low-step inference regimes compared to baseline diffusion models?

Key findings

• AnimateLCM achieves state-of-the-art performance in FVD and CLIPSIM metrics, especially in the 1–4 step inference regime, outperforming baselines even without classifier-free guidance (CFG).
• The decoupled consistency learning strategy significantly improves training convergence and final generation quality, outperforming both 'w/o decouple' and 'w/o init' baselines in quantitative ablation.
• The teacher-free adapter adaptation strategy enables stable, high-quality controllable generation with T2I-Adapter, reducing flickering and improving control accuracy compared to direct adapter usage.
• Replacing the spatial weights with a personalized realistic style model (Realistic Vision V5.0) boosts both visual quality and motion smoothness, demonstrating strong compatibility with personalized models.
• The method reduces inference time and peak memory cost by 50% compared to baselines using 7.5 CFG strength, due to the absence of CFG during evaluation.