[Paper Review] Fourier Diffusion Models: A Method to Control MTF and NPS in Score-Based Stochastic Image Generation

Matthew Tivnan, Jacopo Teneggi+7 more|arXiv (Cornell University)|Mar 23, 2023

Generative Adversarial Networks and Image Synthesis4 citations

TL;DR

This paper introduces Fourier Diffusion Models, a novel score-based generative modeling approach that replaces scalar diffusion processes with shift-invariant convolutions and stationary noise, enabling direct control of MTF and NPS at each time step. The method achieves superior image generation performance with significantly fewer time steps—matching scalar models at 128 or 1024 steps using only 8 steps—by modeling continuous probability flow from true images to measured, blurred, and noisy images with known system characteristics.

ABSTRACT

Score-based stochastic denoising models have recently been demonstrated as powerful machine learning tools for conditional and unconditional image generation. The existing methods are based on a forward stochastic process wherein the training images are scaled to zero over time and white noise is gradually added such that the final time step is approximately zero-mean identity-covariance Gaussian noise. A neural network is then trained to approximate the time-dependent score function, or the gradient of the logarithm of the probability density, for that time step. Using this score estimator, it is possible to run an approximation of the time-reversed stochastic process to sample new images from the training data distribution. These score-based generative models have been shown to out-perform generative adversarial neural networks using standard benchmarks and metrics. However, one issue with this approach is that it requires a large number of forward passes of the neural network. Additionally, the images at intermediate time steps are not useful, since the signal-to-noise ratio is low. In this work we present a new method called Fourier Diffusion Models which replaces the scalar operations of the forward process with shift-invariant convolutions and the additive white noise with additive stationary noise. This allows for control of MTF and NPS at intermediate time steps. Additionally, the forward process can be crafted to converge to the same MTF and NPS as the measured images. This way, we can model continuous probability flow from true images to measurements. In this way, the sample time can be used to control the tradeoffs between measurement uncertainty and generative uncertainty of posterior estimates. We compare Fourier diffusion models to existing scalar diffusion models and show that they achieve a higher level of performance and allow for a smaller number of time steps.

Motivation & Objective

• To address the inefficiency of conventional score-based diffusion models that require many time steps due to degradation to pure white noise.
• To enable direct control of modulation transfer function (MTF) and noise power spectrum (NPS) during the forward and reverse diffusion processes.
• To improve image generation performance by aligning the forward process with real imaging system characteristics, such as spatial blur and textured noise.
• To reduce computational cost by enabling high-quality generation with fewer reverse sampling steps.
• To demonstrate the method's effectiveness in low-radiation-dose CT image restoration, a clinically relevant application.

Proposed method

• Replace scalar forward diffusion with shift-invariant convolutional operations using circulant matrices to model system blur (MTF) in the Fourier domain.
• Model additive noise as stationary Gaussian noise with a specified noise power spectrum (NPS), enabling frequency-domain control of noise characteristics.
• Define the forward process as a sequence of LSI system responses with ASGN, allowing the final state to match measured image statistics rather than pure noise.
• Train a score-matching neural network to estimate the time-dependent score function in the Fourier domain, enabling iterative denoising and sharpening.
• Perform reverse sampling using the learned score function to generate high-quality images from measured data, with time steps representing progression from noisy, blurred measurements to clean images.
• Use a generalization of the score-matching loss and reverse SDE formulation applicable to any Gaussian process with linear systems and additive Gaussian noise, not limited to shift-invariant systems.

Figure 1: A probabilistic graphical model for the stochastic process, consisting of linear shift invariant systems and additive stationary Gaussian noise. The measurement-conditioned diffusion model, in light purple, is trained to approximate the reverse process, in dark purple.

Experimental results

Research questions

• RQ1 Can controlling MTF and NPS during the forward diffusion process improve the efficiency and quality of score-based image generation?
• RQ2 Does modeling the forward process to converge to measured image statistics (rather than pure noise) reduce the number of reverse sampling steps required for high-quality generation?
• RQ3 How does the similarity between initial measured images and the final state of the forward process affect the performance of the reverse sampling process?
• RQ4 Can Fourier Diffusion Models outperform scalar diffusion models in image restoration tasks with fewer time steps and lower computational cost?
• RQ5 Is the improved performance due to reduced distribution shift between the reverse process start and the true data distribution, or due to architectural compatibility with convolutional networks?

Key findings

• Fourier Diffusion Models achieve comparable or better image quality than scalar diffusion models using only 8 time steps, matching the performance of scalar models at 128 or 1024 steps.
• The mean squared error (MSE) of Fourier diffusion models is significantly lower than scalar models across all time step counts, with the largest improvement observed at low time steps.
• The mean squared bias and mean variance metrics also show consistent improvement for Fourier diffusion models, indicating better fidelity and reduced estimation error.
• Visual comparisons demonstrate that Fourier diffusion models produce higher-quality reconstructions, especially at low time steps, with reduced artifacts and improved structural preservation.
• The reverse process in Fourier diffusion models shows less degradation over time compared to scalar models, suggesting more stable and accurate score estimation.
• The results suggest that the improved performance stems from the closer distributional similarity between the initial measured image and the final forward state, reducing the burden on the neural network to invert artificial degradation processes.

Figure 2: Diagram of the score-matching neural network. The inputs are the forward process sample image, low-dose CT measured image, and sample time. The output is the estimated score function.