[Paper Review] Feeding the zombies: Synthesizing brain volumes using a 3D progressive growing GAN

Anders Eklund|arXiv (Cornell University)|Dec 11, 2019

Generative Adversarial Networks and Image Synthesis24 references5 citations

TL;DR

This paper proposes a 3D progressive growing GAN (3D PGAN) to synthesize high-fidelity T1-weighted MR brain volumes from a limited dataset. By training progressively from 4×4×4 to 64×64×64 voxel resolutions using data augmentation and 3D convolutions, the model generates realistic brain volumes that can augment training data for medical AI, demonstrating feasibility for 3D medical image synthesis with minimal mode collapse.

ABSTRACT

Deep learning requires large datasets for training (convolutional) networks with millions of parameters. In neuroimaging, there are few open datasets with more than 100 subjects, which makes it difficult to, for example, train a classifier to discriminate controls from diseased persons. Generative adversarial networks (GANs) can be used to synthesize data, but virtually all research is focused on 2D images. In medical imaging, and especially in neuroimaging, most datasets are 3D or 4D. Here we therefore present preliminary results showing that a 3D progressive growing GAN can be used to synthesize MR brain volumes.

Motivation & Objective

• To address the scarcity of large-scale open neuroimaging datasets by generating synthetic brain volumes.
• To adapt the 2D progressive growing GAN framework to 3D medical imaging for improved training stability and image quality.
• To evaluate whether a 3D PGAN can generate high-resolution, realistic T1-weighted MR brain volumes from limited real data.
• To enable data augmentation and privacy-preserving data sharing through synthetic volume generation.
• To provide a reproducible 3D GAN framework for future medical image synthesis research.

Proposed method

• The method employs a 3D progressive growing GAN trained progressively from 4×4×4 to 64×64×64 voxel resolutions, starting with low-resolution volumes and gradually increasing complexity.
• All 2D convolutions in the original 2D PGAN implementation were replaced with 3D convolutions, and TensorFlow operations were extended to handle 3D tensors.
• Data augmentation was applied by generating 10 random 3D rotations per subject, increasing the training set from 900 to 9,000 volumes with rotations drawn from a normal distribution (mean 0°, std 10°).
• The model uses a noise vector as input to the generator, which produces synthetic volumes, while the discriminator distinguishes real from generated volumes through adversarial training.
• NIfTI file I/O was implemented using the nibabel Python package to handle neuroimaging data formats.
• Hyperparameters, including learning rates and minibatch sizes, were adjusted per resolution level, with reduced filter counts (max 128) compared to the original 2D PGAN.

Experimental results

Research questions

• RQ1 Can a 3D progressive growing GAN effectively synthesize realistic T1-weighted MR brain volumes at high resolution?
• RQ2 Does progressive training from low to high resolution improve training stability and image quality in 3D medical GANs?
• RQ3 To what extent can data augmentation via 3D rotations improve generalization and reduce mode collapse in 3D GANs for neuroimaging?
• RQ4 Can synthetic brain volumes generated by a 3D PGAN be used to effectively augment real datasets for downstream medical AI tasks?
• RQ5 How does the performance of a 3D PGAN compare to existing 2D or small 3D GANs in synthesizing full-brain MR volumes?

Key findings

• The 3D PGAN successfully generated realistic T1-weighted MR brain volumes at 64×64×64 voxel resolution, demonstrating the feasibility of 3D progressive training for medical imaging.
• Synthetic volumes at 32×32×32 and 64×64×64 resolutions showed coherent anatomical structures, including sulcal and gyral patterns, indicating plausible brain morphology.
• Training time for the full 64×64×64 resolution model was approximately 81 hours on an Nvidia DGX station with four V100 GPUs.
• The model achieved stable training with minimal mode collapse, as evidenced by diverse and anatomically plausible outputs across multiple random seeds.
• The use of data augmentation with 10 random 3D rotations per subject (total 9,000 volumes) helped improve training stability and diversity of generated samples.
• The 3D PGAN framework was released on GitHub, enabling reproducibility and future extension to higher resolutions or other anatomical structures.