[Paper Review] High-resolution medical image synthesis using progressively grown generative adversarial networks

Andrew Beers, James M. Brown+5 more|arXiv (Cornell University)|May 8, 2018

Cell Image Analysis TechniquesBiochemistry, Genetics and Molecular Biology49 citations

TL;DR

The paper applies progressively grown GANs (PGGANs) to create high-resolution medical images in retinal fundus and multi-modal glioma MRI, showing that segmentation channels improve pathology details and that latent space encodes clinically relevant information.

ABSTRACT

Generative adversarial networks (GANs) are a class of unsupervised machine learning algorithms that can produce realistic images from randomly-sampled vectors in a multi-dimensional space. Until recently, it was not possible to generate realistic high-resolution images using GANs, which has limited their applicability to medical images that contain biomarkers only detectable at native resolution. Progressive growing of GANs is an approach wherein an image generator is trained to initially synthesize low resolution synthetic images (8x8 pixels), which are then fed to a discriminator that distinguishes these synthetic images from real downsampled images. Additional convolutional layers are then iteratively introduced to produce images at twice the previous resolution until the desired resolution is reached. In this work, we demonstrate that this approach can produce realistic medical images in two different domains; fundus photographs exhibiting vascular pathology associated with retinopathy of prematurity (ROP), and multi-modal magnetic resonance images of glioma. We also show that fine-grained details associated with pathology, such as retinal vessels or tumor heterogeneity, can be preserved and enhanced by including segmentation maps as additional channels. We envisage several applications of the approach, including image augmentation and unsupervised classification of pathology.

Motivation & Objective

• Demonstrate that PGGANs can generate high-resolution medical images (up to 512x512) in clinically relevant domains.
• Show that including segmentation maps as additional channels preserves fine-grained pathological details.
• Assess the realism and diversity of synthetic medical images and their potential for augmentation and unsupervised learning.
• Explore the latent space of PGGANs to determine whether clinically meaningful features are encoded.

Proposed method

• Use progressively growing of GANs to train a generator and discriminator that incrementally increase image resolution from 4x4 up to 512x512 (or 256x256 for MRI).
• Train with Wasserstein loss and equal generator-discriminator updates.
• Include segmentation maps as additional channels to guide synthesis of diagnostically relevant structures.
• Encode real images into latent space with a separate network to study latent representations and perform nearest-neighbor in latent space.
• Evaluate image quality qualitatively and via a segmentation algorithm on synthetic fundus images (AUC reported as 0.97 for vessel segmentation reference).
• Preprocess data with domain-specific steps (retinal vasculature segmentation; BraTS MR modalities T1, T1, T2, FLAIR).

Experimental results

Research questions

• RQ1 Can high-resolution medical images be synthesized convincingly using progressively grown GANs in retinal fundus and glioma MRI data?
• RQ2 Does incorporating segmentation maps as additional channels improve preservation of diagnostically relevant structures?
• RQ3 Is the latent space of the PGGAN encoding clinically meaningful, enabling unsupervised feature learning and nearest-neighbor retrieval?
• RQ4 What is the diversity and realism of synthetic images across modalities and pathologies?
• RQ5 Can synthetic data contribute to image augmentation or cross-modality synthesis in multi-institution biomedical imaging?

Key findings

• PGGANs produced high-quality, varied synthetic retinal fundus and multi-modal MRI images at high resolutions.
• Segmentation-channel augmentation improved visualization of pathology and preserved diagnostically relevant details.
• Synthetic vessel maps could be used by a segmentation algorithm to achieve high alignment with real vessel structures (AUC ~0.97 in tested setup).
• Latent space encodings revealed interpretable structure, such as right vs left eye and axial progression through brain slices in latent/encoded representations.
• Encoding real images into the latent space allowed approximate nearest-neighbor retrieval, indicating potential for unsupervised feature learning and data augmentation.