[Paper Review] Brain Imaging Generation with Latent Diffusion Models

Walter Hugo Lopez Pinaya, Petru-Daniel Tudosiu+6 more|arXiv (Cornell University)|Sep 15, 2022

Radiomics and Machine Learning in Medical ImagingMedicine26 citations

TL;DR

The paper trains latent diffusion models (LDMs) to generate high-resolution 3D brain MRIs conditioned on covariates, evaluating with FID and diversity metrics, and releases a 100k-sample synthetic brain dataset.

ABSTRACT

Deep neural networks have brought remarkable breakthroughs in medical image analysis. However, due to their data-hungry nature, the modest dataset sizes in medical imaging projects might be hindering their full potential. Generating synthetic data provides a promising alternative, allowing to complement training datasets and conducting medical image research at a larger scale. Diffusion models recently have caught the attention of the computer vision community by producing photorealistic synthetic images. In this study, we explore using Latent Diffusion Models to generate synthetic images from high-resolution 3D brain images. We used T1w MRI images from the UK Biobank dataset (N=31,740) to train our models to learn about the probabilistic distribution of brain images, conditioned on covariables, such as age, sex, and brain structure volumes. We found that our models created realistic data, and we could use the conditioning variables to control the data generation effectively. Besides that, we created a synthetic dataset with 100,000 brain images and made it openly available to the scientific community.

Motivation & Objective

• Motivate synthetic data generation to overcome limited medical imaging datasets and privacy concerns.
• Develop high-resolution 3D brain image generation using Latent Diffusion Models to scale with large datasets.
• Enable conditioning on covariates (age, sex, ventricular and brain volumes) to produce realistic, controllable brain images.
• Compare LDMs to GAN-based baselines and assess realism and diversity of generated images.
• Provide open access to a large synthetic brain MRI dataset for the community.

Proposed method

• Use UK Biobank T1w MRI (N=31,740) to train a latent diffusion model on 20×28×20 latent representations.
• Compress images with an autoencoder and train diffusion in latent space with 1000-step forward process and a fixed variance schedule.
• Condition image generation on age, sex, ventricular volume, and brain volume via concatenation and cross-attention (hybrid conditioning).
• Evaluate sampling quality with FID (Med3D features) and diversity with MS-SSIM and 4-G-R-SSIM; compare against LSGAN and VAE-GAN baselines.
• Apply DDIM to speed sampling, reducing steps from 1000 to 50 with minimal performance loss.

Experimental results

Research questions

• RQ1 Can latent diffusion models generate high-resolution 3D brain MRIs that match real data distributions?
• RQ2 How well can conditioning variables (age, sex, ventricular and brain volumes) control generated brain images?
• RQ3 Do LDMs outperform GAN-based baselines in realism and diversity for 3D brain MRI synthesis?
• RQ4 Is it feasible to scale high-resolution brain image generation using latent representations?
• RQ5 What is the utility and impact of releasing a large synthetic brain MRI dataset for the community?

Key findings

• LDMs produced high-quality brain MRIs with sharp details and realistic textures, outperforming GAN baselines in unconditioned generation.
• DDIM sampling significantly speeds up generation (50 steps vs 1000) with minimal accuracy loss.
• conditioning effectively controlled ventricular and brain volumes, with a high correlation (r = 0.972) between input and SynthSeg-measured ventricles.
• Brain age conditioning showed a strong correlation between input conditioning and predicted age (r = 0.692).
• Extrapolations of conditioning variables demonstrated learned representations, e.g., larger ventricles or signs of neurodegeneration when ventricle/brain size values were outside training range.
• A synthetic dataset of 100,000 brain images was released publicly for research use.