[Paper Review] DreamPose: Fashion Image-to-Video Synthesis via Stable Diffusion

Johanna Karras, Aleksander Holynski+2 more|arXiv (Cornell University)|Apr 12, 2023

Generative Adversarial Networks and Image Synthesis4 citations

TL;DR

DreamPose is a diffusion-based method that generates photorealistic fashion videos from a single still image and a sequence of human body poses by adapting Stable Diffusion with pose and image conditioning. It achieves state-of-the-art results in temporal consistency, appearance fidelity, and motion realism through a dual CLIP-VAE encoder, 5-pose conditioning, and subject-specific fine-tuning of the UNet and VAE decoder.

ABSTRACT

We present DreamPose, a diffusion-based method for generating animated fashion videos from still images. Given an image and a sequence of human body poses, our method synthesizes a video containing both human and fabric motion. To achieve this, we transform a pretrained text-to-image model (Stable Diffusion) into a pose-and-image guided video synthesis model, using a novel fine-tuning strategy, a set of architectural changes to support the added conditioning signals, and techniques to encourage temporal consistency. We fine-tune on a collection of fashion videos from the UBC Fashion dataset. We evaluate our method on a variety of clothing styles and poses, and demonstrate that our method produces state-of-the-art results on fashion video animation.Video results are available on our project page.

Motivation & Objective

• To address the lack of realistic, temporally consistent fashion videos despite the prevalence of fashion photography.
• To enable high-fidelity image animation with precise control over human motion and fabric dynamics using only a single input image and pose sequence.
• To overcome limitations of existing video diffusion models, such as poor temporal consistency and reliance on text-only conditioning.
• To improve appearance fidelity and motion smoothness in fashion video synthesis by integrating pose and image conditioning into a pretrained diffusion model.
• To achieve subject-specific animation without requiring extensive data or complex multi-stage networks.

Proposed method

• Fine-tunes a pretrained text-to-image Stable Diffusion model using a novel two-stage strategy: first on the UBC Fashion dataset, then with subject-specific fine-tuning on a single input image.
• Replaces the CLIP text encoder with a dual CLIP-VAE image encoder and adapter module to jointly model and reshape image and latent embeddings for improved appearance fidelity.
• Conditions the denoising U-Net on a sequence of five consecutive poses centered on the target frame, enhancing temporal consistency.
• Introduces a novel architecture that supports both image and pose conditioning by modifying the attention mechanisms and input injection pathways.
• Performs subject-specific fine-tuning of the UNet, adapter module, and VAE decoder to preserve identity and texture details across frames.
• Employs a noise injection mechanism where the pose representation is concatenated to the input noise at each denoising step.

Figure 2: Architecture Overview. We modify the original Stable Diffusion architecture in order to enable image and pose conditioning. First, we replace the CLIP text encoder with a dual CLIP-VAE image encoder and adapter module (shown in the blue box). The adapter module jointly models and reshapes

Experimental results

Research questions

• RQ1 Can a pretrained image diffusion model be effectively adapted to generate high-quality, photorealistic fashion videos from a single image and pose sequence?
• RQ2 How does pose conditioning with multiple sequential poses improve temporal consistency compared to single-pose conditioning?
• RQ3 To what extent does subject-specific fine-tuning of both the UNet and VAE decoder enhance appearance fidelity and reduce flickering?
• RQ4 How does the dual CLIP-VAE encoder compare to a standard CLIP image encoder in preserving fine details of clothing and identity?
• RQ5 What are the failure modes of the method, and can they be mitigated with improved pose estimation or additional supervision?

Key findings

• The full DreamPose model achieves the best performance across all metrics: L1 (0.019), SSIM (0.900), VGG (0.207), and LPIPS (0.056), outperforming ablated variants.
• The ablation with only a CLIP image encoder (Ours_CLIP) shows significant appearance degradation, indicating the necessity of the dual CLIP-VAE encoder for detail preservation.
• Subject-specific fine-tuning of the VAE decoder significantly improves sharpness and reduces overfitting, as shown by the improved metrics in Ours_No-VAE-FT compared to Ours_CLIP.
• Using only one pose (Ours_1-pose) leads to flickering and motion instability, despite similar quantitative scores, highlighting the importance of multi-frame pose context.
• Temporal smoothing applied to the 1-pose variant (Ours_smooth) drastically degrades performance, indicating that motion artifacts are better resolved by architectural design than post-processing.
• Failure cases include limb merging into fabric, hallucinated features, and directional misalignment in back-facing poses, suggesting limitations in pose estimation and geometric reasoning.

Figure 3: Qualitative Results. We showcase the results of our method on a variety of input frames and poses. DreamPose is capable of synthesizing photorealistic video frames consistent with a diverse range of patterns, fabric types, person identities, clothing shapes, and viewpoints.