[Paper Review] Diffusion Alignment Beyond KL: Variance Minimisation as Effective Policy Optimiser

Zijing Ou, Jacob Si+5 more|arXiv (Cornell University)|Feb 12, 2026

Functional Brain Connectivity Studies0 citations

TL;DR

VMPO reframes diffusion alignment as variance minimisation of log importance weights, linking to KL-based methods while enabling new design directions; empirically improves reward-based alignment on Stable Diffusion.

ABSTRACT

Diffusion alignment adapts pretrained diffusion models to sample from reward-tilted distributions along the denoising trajectory. This process naturally admits a Sequential Monte Carlo (SMC) interpretation, where the denoising model acts as a proposal and reward guidance induces importance weights. Motivated by this view, we introduce Variance Minimisation Policy Optimisation (VMPO), which formulates diffusion alignment as minimising the variance of log importance weights rather than directly optimising a Kullback-Leibler (KL) based objective. We prove that the variance objective is minimised by the reward-tilted target distribution and that, under on-policy sampling, its gradient coincides with that of standard KL-based alignment. This perspective offers a common lens for understanding diffusion alignment. Under different choices of potential functions and variance minimisation strategies, VMPO recovers various existing methods, while also suggesting new design directions beyond KL.

Motivation & Objective

• Motivate diffusion alignment to steer pretrained diffusion models toward high-reward samples.
• Introduce Variance Minimisation Policy Optimisation (VMPO) as an alternative to KL-based objectives.
• Show that variance minimisation yields the same gradient as KL under on-policy sampling.
• Demonstrate that VMPO recovers existing methods under certain choices and enables new design directions.
• Empirically validate VMPO by finetuning Stable Diffusion 1.5 and 3.5 across diverse rewards.

Proposed method

• Treat the denoising process as a sequential proposal in a Sequential Monte Carlo view.
• Define the VMPO objective as minimising the variance of log importance weights along the trajectory (Eq. 4).
• Prove that the optimum yields the reward-tilted target and that on-policy gradients coincide with KL-based alignment (Proposition 1).
• Estimate the VMPO loss via Monte Carlo samples and introduce a neural estimator M_phi to amortise the log-weight expectation (Eq. 8–9).
• Derive training procedure and instantiate two variants VMPO-R2G and VMPO-Diff through different reward potentials (Appendix C).
• Show that VMPO connects to GRPO and other diffusion-alignment methods as special cases under specific variance strategies (Appendix C).

Experimental results

Research questions

• RQ1 How can diffusion alignment be formulated beyond KL minimisation?
• RQ2 Does variance minimisation yield equivalent gradients to KL under on-policy sampling and what are the practical benefits?
• RQ3 How do different potential functions and variance strategies relate to existing diffusion-alignment methods?
• RQ4 Can VMPO improve reward-driven alignment when finetuning real diffusion models on practical reward signals?

Key findings

• VMPO optimises diffusion alignment by minimising the variance of log importance weights along the denoising trajectory.
• Under on-policy sampling, the VMPO gradient matches the gradient of KL-based alignment.
• VMPO with different variance strategies recovers existing methods and suggests new design directions beyond KL.
• Empirically, VMPO improves human-preference-based alignment (HPSv2) and ImageReward on Stable Diffusion 1.5, with VMPO-Diff achieving the strongest reward signal.
• VMPO-Diff increases reward while incurring trade-offs in CLIPScore and DreamSim, indicating reward hacking tendencies similar to other methods.
• The paper provides a unified probabilistic lens (SMC) for understanding diffusion alignment and its variants.