[Paper Review] Making Reconstruction FID Predictive of Diffusion Generation FID

Tongda Xu, Mingwei He+7 more|arXiv (Cornell University)|Mar 5, 2026

Advanced Neuroimaging Techniques and Applications0 citations

TL;DR

The paper introduces interpolated FID (iFID), a simple latent-space interpolation metric that strongly correlates with diffusion model generation FID (gFID), addressing the reconstruction–generation dilemma. It shows rFID correlates with refinement-phase quality while iFID aligns with navigation-phase quality, and provides explanations based on diffusion generalization and hallucination, with code released.

ABSTRACT

It is well known that the reconstruction FID (rFID) of a VAE is poorly correlated with the generation FID (gFID) of a latent diffusion model. We propose interpolated FID (iFID), a simple variant of rFID that exhibits a strong correlation with gFID. Specifically, for each element in the dataset, we retrieve its nearest neighbor (NN) in the latent space and interpolate their latent representations. We then decode the interpolated latent and compute the FID between the decoded samples and the original dataset. Additionally, we refine the claim that rFID correlates poorly with gFID, by showing that rFID correlates with sample quality in the diffusion refinement phase, whereas iFID correlates with sample quality in the diffusion navigation phase. Furthermore, we provide an explanation for why iFID correlates well with gFID, and why reconstruction metrics are negatively correlated with gFID, by connecting to results in the diffusion generalization and hallucination. Empirically, iFID is the first metric to demonstrate a strong correlation with diffusion gFID, achieving Pearson linear and Spearman rank correlations approximately 0.85. The source code is provided in https://github.com/tongdaxu/Making-rFID-Predictive-of-Diffusion-gFID.

Motivation & Objective

• Motivate the need for a metric that predicts diffusion generation quality from VAE reconstructions.
• Propose a simple latent-space interpolation variant of rFID (iFID) and demonstrate its strong correlation with gFID.
• Refine the understanding of how rFID relates to diffusion sample quality across refinement and navigation phases.
• Explain why iFID correlates with diffusion performance and why standard reconstruction metrics may fail.
• Evaluate iFID across diverse VAEs and diffusion models on ImageNet.

Proposed method

• Define rFID and gFID in the latent-diffusion setting (VAE encoder, decoder g, diffusion solver Φ).
• Introduce iFID as FID between original images and decoded interpolated latent ẑ = 0.5(z + NN(z)) where NN(z) is the nearest neighbor in latent space.
• Assess correlations (PCC and SRCC) between rFID/iFID/gFID across diffusion trajectories and phases.
• Perform ablations on interpolation type (linear, spherical, mask), interpolation strength α, and nearest-neighbor set size to test robustness.
• Analyze why iFID tracks diffusion quality by connecting to diffusion generalization/hallucination literature.
• Compare iFID with reconstruction metrics and non-reconstruction losses (Diffusion Loss, EQ/SE/VF/GMM Loss).

Figure 1: Left two plots : The rFID values of VAEs are uncorrelated, or even negatively correlated with, the gFID values of diffusion models. Right two plots : iFID metric exhibits a strong positive correlation with the gFID values of diffusion models.

Experimental results

Research questions

• RQ1 Can iFID provide a stronger, more reliable proxy for diffusion gFID than rFID across VAEs?
• RQ2 How do rFID and iFID relate to diffusion sample quality in the refinement vs. navigation phases?
• RQ3 Why does iFID correlate with gFID in terms of interpolation of training data and latent space structure?
• RQ4 What latent-space properties (connectivity, interpolation validity) influence diffusion generation quality?

Key findings

• iFID shows a strong correlation with diffusion gFID (≈0.85 Pearson and Spearman) across models and settings.
• rFID correlates with diffusion sample quality in the refinement phase, while iFID correlates with quality in the navigation phase.
• Reconstruction metrics (PSNR, SSIM, LPIPS) negatively correlate with diffusion gFID, illustrating the reconstruction–generation dilemma.
• iFID outperforms non-reconstruction metrics and diffusion losses in predicting gFID, indicating it captures the validity of interpolated latent representations.
• Robustness analyses show iFID is stable to interpolation method (linear, spherical, mask), subset size for NN(z), and top-K neighbor usage; spherical interpolation yields the highest correlation.
• The authors provide intuition linking iFID to diffusion generalization and hallucination, explaining why interpolations in latent space reflect generation performance.

Figure 2: The refinement and navigation phases are key components of the sampling process for SiT-XL trained with SD-VAE. In the refinement phase (small $t$ ), the sample generated from the noisy source is nearly identical to the source. In contrast, during the navigation phase (large $t$ ), the sam