[Paper Review] Generative Adversarial Network for Image Synthesis

Yang Lei, Richard L. J. Qiu+4 more|arXiv (Cornell University)|Dec 31, 2020

Generative Adversarial Networks and Image Synthesis82 references4 citations

TL;DR

This paper reviews conditional GAN (cGAN) and Cycle-GAN architectures for medical and biomedical image synthesis, demonstrating their ability to generate high-fidelity synthetic images from source modalities like MRI or CT. The key contribution is a systematic analysis of network designs, clinical applications, and challenges—particularly the reliance on paired data in cGAN and the trade-offs in performance with unpaired Cycle-GAN.

ABSTRACT

This chapter reviews recent developments of generative adversarial networks (GAN)-based methods for medical and biomedical image synthesis tasks. These methods are classified into conditional GAN and Cycle-GAN according to the network architecture designs. For each category, a literature survey is given, which covers discussions of the network architecture designs, highlights important contributions and identifies specific challenges.

Motivation & Objective

• To review recent advances in generative adversarial networks (GANs) for medical and biomedical image synthesis.
• To classify and analyze cGAN and Cycle-GAN architectures based on their design principles and performance in image synthesis tasks.
• To highlight key contributions and identify persistent challenges in clinical deployment, such as paired data dependency and generalization to rare anatomical cases.
• To evaluate the performance of GAN-based methods against conventional model-based approaches in generating realistic synthetic images.
• To explore the potential of GANs in clinical applications including MRI-only radiotherapy planning, image denoising, and synthetic data generation.

Proposed method

• The study employs a literature review methodology focusing on peer-reviewed journal and conference publications from PubMed, using keyword criteria related to image synthesis, deep learning, GANs, and medical imaging modalities.
• Network architectures are categorized into conditional GAN (cGAN) and Cycle-GAN based on their training paradigm: cGAN uses paired source-target image data, while Cycle-GAN operates on unpaired data using cycle-consistency loss.
• The cGAN framework involves a generator network that learns to map source images to target images, and a discriminator that distinguishes real from generated images through adversarial training.
• Cycle-GAN enforces cycle consistency by training the generator to reconstruct the original image after a two-step transformation (X → Y → X), reducing reliance on paired data.
• Performance evaluation is based on qualitative realism and quantitative metrics such as structural similarity index (SSIM), peak signal-to-noise ratio (PSNR), and mean squared error (MSE), though specific values are not reported in the paper.
• The analysis includes clinical applications such as MRI-only treatment planning, PET/MRI synthesis, proton stopping power estimation, and image quality enhancement using GANs.

Experimental results

Research questions

• RQ1 How do cGAN and Cycle-GAN architectures compare in generating realistic synthetic medical images across different imaging modalities?
• RQ2 What are the key architectural innovations in cGAN and Cycle-GAN that enable high-fidelity image synthesis in medical imaging?
• RQ3 To what extent do GAN-based methods outperform conventional model-based methods in image synthesis tasks such as denoising, super-resolution, and image reconstruction?
• RQ4 What are the major challenges in deploying GANs clinically, particularly regarding data pairing requirements and robustness to rare or artifact-prone anatomical cases?
• RQ5 How do GAN-based methods handle generalization to out-of-distribution inputs such as patients with implants, obesity, or anatomical abnormalities?

Key findings

• GAN-based methods, particularly cGAN and Cycle-GAN, generate more realistic synthetic images than conventional model-based approaches, with improved visual fidelity and quantitative similarity to real images.
• cGANs require paired training data with pixel-wise correspondence, which poses challenges in data collection and image registration accuracy.
• Cycle-GAN relaxes the need for paired data by enforcing cycle consistency, enabling training on unpaired datasets and improving scalability for large clinical cohorts.
• Despite better image quality, Cycle-GAN may not significantly improve numerical metrics like PSNR or SSIM compared to cGAN in some synthesis tasks due to residual distribution mismatches.
• GAN-based models generalize poorly to out-of-distribution cases such as patients with hip prostheses, implants, or severe artifacts, which remain under-investigated in current literature.
• Once trained, GAN models can generate synthetic images in seconds or minutes, making them practical for clinical deployment despite lengthy training times of hours to days.