[Paper Review] Interpolating between Images with Diffusion Models

Clinton Wang, Polina Golland|arXiv (Cornell University)|Jul 24, 2023

Generative Adversarial Networks and Image SynthesisComputer Science3 citations

TL;DR

This paper proposes a zero-shot image interpolation method using pre-trained latent diffusion models, enabling high-quality, semantically meaningful transitions between real images with diverse styles, layouts, and subjects. By interpolating in latent space at decreasing noise levels and conditioning on interpolated text embeddings and optional pose guidance, the method produces visually coherent sequences that outperform standard metrics like FID and PPL, which favor simple alpha blending over creative semantic transformations.

ABSTRACT

One little-explored frontier of image generation and editing is the task of interpolating between two input images, a feature missing from all currently deployed image generation pipelines. We argue that such a feature can expand the creative applications of such models, and propose a method for zero-shot interpolation using latent diffusion models. We apply interpolation in the latent space at a sequence of decreasing noise levels, then perform denoising conditioned on interpolated text embeddings derived from textual inversion and (optionally) subject poses. For greater consistency, or to specify additional criteria, we can generate several candidates and use CLIP to select the highest quality image. We obtain convincing interpolations across diverse subject poses, image styles, and image content, and show that standard quantitative metrics such as FID are insufficient to measure the quality of an interpolation. Code and data are available at https://clintonjwang.github.io/interpolation.

Motivation & Objective

• To enable high-quality, semantically meaningful interpolation between real images with large differences in style, content, and layout, a capability missing in current image generation pipelines.
• To address the lack of user control and consistency in existing interpolation methods, especially for real images outside of limited domains like faces.
• To develop a method that leverages pre-trained latent diffusion models without fine-tuning, enabling zero-shot deployment across diverse image pairs.
• To demonstrate that standard metrics like FID and PPL fail to capture perceptual quality and creativity in interpolations, necessitating new evaluation paradigms.

Proposed method

• Interpolate latent representations of two input images at multiple noise levels in the diffusion process, using a uniform or non-uniform schedule to control perceptual transition speed.
• Condition the denoising U-Net on interpolated text embeddings derived via textual inversion, enabling semantic consistency across frames.
• Optionally incorporate subject pose embeddings to maintain structural coherence during transformations, especially for human or anthropomorphic subjects.
• Generate multiple candidate interpolations per frame using different noise vectors and select the highest quality output using CLIP similarity to a desired prompt.
• Use a multi-stage denoising strategy—interpolating at intermediate timesteps and denoising iteratively—to improve smoothness and semantic fidelity.
• Apply affine transformations to latents before interpolation to simulate motion effects like zooming or panning, extending the method to video-like sequences.

Figure 2: Our pipeline. To generate a new frame, we interpolate the noisy latent images of two existing frames (Section 4.1 ). Text prompts and (if applicable) poses are extracted from the original input images, and interpolated to provide to the denoiser as conditioning inputs (Section 4.2 and 4.3

Experimental results

Research questions

• RQ1 Can latent diffusion models generate high-quality, semantically coherent interpolations between real images with diverse styles and layouts without fine-tuning?
• RQ2 How does conditioning on interpolated text and pose embeddings affect the quality and consistency of image interpolations?
• RQ3 Why do standard metrics like FID and PPL fail to correlate with human preference in image interpolation tasks?
• RQ4 Can multi-stage denoising and CLIP-based candidate selection improve the visual quality and coherence of interpolations?
• RQ5 How does the choice of interpolation schedule (uniform vs. non-uniform) affect perceptual smoothness and user perception of transitions?

Key findings

• The proposed method produces visually convincing interpolations across diverse image pairs, including transitions from human to mountain, cartoon to photorealistic, and different poses, even with large semantic differences.
• Interpolation using shared noise and multi-stage denoising (Add noise-interpolate-denoise) achieved the highest PPL (193 ± 27) and was preferred qualitatively, despite not being optimal on FID.
• FID and PPL metrics favored simple alpha blending-like interpolations, which were less creative and more abrupt, indicating that current metrics are insufficient for evaluating interpolation quality.
• CLIP-based candidate selection significantly improved output quality by filtering out low-similarity or inconsistent frames, enhancing semantic coherence.
• Non-uniform interpolation schedules can better control perceptual speed, especially for rapid stylistic changes near input images, improving user control over transition dynamics.
• The method occasionally fails to generalize to extreme style mismatches or complex object reorganization, and may insert spurious text or misalign body parts, even with pose guidance.

Figure 3: Pose conditioning mitigates the occurrence of abrupt pose changes between adjacent frames, even when the predicted pose is incorrect.