[Paper Review] ConsistI2V: Enhancing Visual Consistency for Image-to-Video Generation

Weiming Ren, Harry Yang+5 more|arXiv (Cornell University)|Feb 6, 2024

Advanced Vision and Imaging4 citations

TL;DR

ConsistI2V proposes a diffusion-based method to enhance visual consistency in image-to-video (I2V) generation by leveraging spatiotemporal attention over the first frame and noise initialization from its low-frequency band. The approach achieves state-of-the-art performance on I2V-Bench, significantly improving motion smoothness, object consistency, and layout fidelity compared to prior methods.

ABSTRACT

Image-to-video (I2V) generation aims to use the initial frame (alongside a text prompt) to create a video sequence. A grand challenge in I2V generation is to maintain visual consistency throughout the video: existing methods often struggle to preserve the integrity of the subject, background, and style from the first frame, as well as ensure a fluid and logical progression within the video narrative. To mitigate these issues, we propose ConsistI2V, a diffusion-based method to enhance visual consistency for I2V generation. Specifically, we introduce (1) spatiotemporal attention over the first frame to maintain spatial and motion consistency, (2) noise initialization from the low-frequency band of the first frame to enhance layout consistency. These two approaches enable ConsistI2V to generate highly consistent videos. We also extend the proposed approaches to show their potential to improve consistency in auto-regressive long video generation and camera motion control. To verify the effectiveness of our method, we propose I2V-Bench, a comprehensive evaluation benchmark for I2V generation. Our automatic and human evaluation results demonstrate the superiority of ConsistI2V over existing methods.

Motivation & Objective

• To address visual inconsistency in image-to-video generation, where models fail to preserve subject, background, and style from the first frame.
• To improve motion smoothness and temporal coherence in generated videos, especially in long-form or complex narratives.
• To develop a robust, generalizable method that enhances consistency without requiring fine-tuning or architectural overhauls.
• To introduce a comprehensive benchmark, I2V-Bench, for evaluating I2V generation models beyond standard metrics.
• To enable practical applications such as autoregressive long video generation and camera motion control through consistent conditioning.

Proposed method

• Introduces spatiotemporal attention mechanisms that condition the video diffusion model on the first frame across both spatial and temporal dimensions to preserve spatial layout and motion consistency.
• Applies cross-frame attention in spatial layers to enforce fine-grained alignment with the first frame’s content and structure.
• Incorporates a local window of first frame features into temporal attention to stabilize motion trajectories and reduce jitter.
• Proposes FrameInit, a noise initialization strategy that uses the low-frequency band of the first frame to align training and inference noise distributions, reducing visual drift.
• Extends the method to support autoregressive long video generation and camera motion control by reusing the same conditioning and initialization mechanisms.
• Employs a multi-scale evaluation framework using I2V-Bench, combining automatic metrics and human evaluation across diverse categories.

Figure 1: Comparison of image-to-video generation results obtained from SEINE (Chen et al., 2023c ) and our ConsistI2V . SEINE shows degenerated appearance and motion as the video progresses, while our result maintains visual consistency. We feed the same first frame to SEINE and ConsistI2V and show

Experimental results

Research questions

• RQ1 Can spatiotemporal conditioning on the first frame significantly improve visual consistency in image-to-video generation?
• RQ2 Does noise initialization from the low-frequency components of the first frame reduce visual discrepancy during video sampling?
• RQ3 How does the proposed method compare to existing I2V models in terms of motion smoothness, object consistency, and layout fidelity?
• RQ4 Can FrameInit be generalized to other video generation paradigms such as autoregressive long video generation?
• RQ5 To what extent does I2V-Bench provide a reliable and comprehensive evaluation of I2V models beyond standard metrics?

Key findings

• ConsistI2V achieves state-of-the-art performance on I2V-Bench, with a 97.77 average score on text-video overall consistency, outperforming SEINE (97.07) and DynamiCrafter (95.89).
• The method improves motion smoothness by 1.87 points on average compared to SEINE, achieving 96.08 on the All category in motion smoothness evaluation.
• Object consistency is enhanced by 3.21 points over SEINE, with ConsistI2V scoring 32.06 on the All category versus SEINE’s 35.43 in the object consistency metric.
• Layout and scene consistency are significantly improved, with ConsistI2V achieving 24.81 on scene consistency (All), outperforming SEINE’s 27.68.
• FrameInit reduces noise discrepancy between training and inference, leading to more stable and consistent video generation, especially in long-form and camera motion scenarios.
• The method generalizes effectively to autoregressive long video generation and camera motion control, demonstrating its extensibility beyond standard I2V tasks.

Figure 2: Our ConsistI2V framework. In our model, we concatenate the first frame latent $z^{1}$ to the input noise and perform first frame conditioning by augmenting the spatial and temporal self-attention operations in the model with the intermediate hidden states $z_{h}^{1}$ . During inference, we