[Paper Review] SkrGAN: Sketching-rendering Unconditional Generative Adversarial Networks for Medical Image Synthesis

Tianyang Zhang, Huazhu Fu+8 more|arXiv (Cornell University)|Aug 6, 2019

Generative Adversarial Networks and Image Synthesis19 references4 citations

TL;DR

SkrGAN is a novel unconditional GAN that decomposes medical image synthesis into two stages: sketch generation followed by color rendering, using a sketch prior to improve structural fidelity. It achieves state-of-the-art performance in synthesizing retinal, X-ray, CT, and MRI images, with significant gains in segmentation accuracy when used for data augmentation.

ABSTRACT

Generative Adversarial Networks (GANs) have the capability of synthesizing images, which have been successfully applied to medical image synthesis tasks. However, most of existing methods merely consider the global contextual information and ignore the fine foreground structures, e.g., vessel, skeleton, which may contain diagnostic indicators for medical image analysis. Inspired by human painting procedure, which is composed of stroking and color rendering steps, we propose a Sketching-rendering Unconditional Generative Adversarial Network (SkrGAN) to introduce a sketch prior constraint to guide the medical image generation. In our SkrGAN, a sketch guidance module is utilized to generate a high quality structural sketch from random noise, then a color render mapping is used to embed the sketch-based representations and resemble the background appearances. Experimental results show that the proposed SkrGAN achieves the state-of-the-art results in synthesizing images for various image modalities, including retinal color fundus, X-Ray, Computed Tomography (CT) and Magnetic Resonance Imaging (MRI). In addition, we also show that the performances of medical image segmentation method have been improved by using our synthesized images as data augmentation.

Motivation & Objective

• To address the limitation of existing GANs in preserving fine-grained anatomical structures like blood vessels and bones in medical image synthesis.
• To introduce a sketch prior that mimics human painting procedures—starting with structural sketching then color rendering—thereby improving generation quality.
• To develop an unconditional GAN that does not require conditional supervision or paired data, such as depth maps, for training.
• To evaluate whether synthesized images from SkrGAN can effectively serve as data augmentation to improve downstream medical image segmentation.

Proposed method

• The method employs a two-stage architecture: a sketch guidance module $G_S$ that generates structural sketches from random noise, guided by a sketch discriminator $D_S$.
• A color render mapping module $G_P$ then generates the final color image by fusing the sketch representation with background appearance modeling, using a pixel-level discriminator $D_P$.
• The sketch guidance module is trained to preserve fine anatomical structures such as vessels and bones, using adversarial loss and perceptual supervision.
• The color rendering module learns to produce realistic textures and intensities by conditioning on the sketch, with adversarial and cycle-consistency losses to ensure fidelity.
• A sketch draft dataset is constructed from real images to supervise the sketch generation stage, enhancing structural accuracy.
• The overall training is end-to-end with joint optimization of both modules via GAN loss, perceptual loss, and structural similarity (MS-SSIM) loss.

Experimental results

Research questions

• RQ1 Can a sketch-guided two-stage GAN architecture improve the fidelity of generated medical images, particularly in preserving fine anatomical structures like blood vessels and bones?
• RQ2 How does SkrGAN compare to state-of-the-art unconditional GANs (e.g., DCGAN, WGAN, PGGAN) in synthesizing diverse medical image modalities?
• RQ3 To what extent can SkrGAN-generated images enhance the performance of downstream medical image segmentation tasks through data augmentation?
• RQ4 Does the use of a sketch prior lead to more stable and controllable image generation compared to direct noise-to-image GANs?

Key findings

• SkrGAN achieved the lowest Fréchet Inception Distance (FID) of 27.59 on retinal fundus images, outperforming DCGAN (64.83), ACGAN (96.72), WGAN (240.7), and PGGAN (110.8).
• On chest X-ray images, SkrGAN achieved an FID of 114.6, significantly lower than DCGAN (260.3), ACGAN (235.2), WGAN (300.7), and PGGAN (124.2).
• For lung CT, SkrGAN achieved an FID of 79.97, outperforming DCGAN (285.0), ACGAN (222.5), WGAN (349.1), and PGGAN (91.89).
• On brain MRI, SkrGAN achieved an FID of 27.51, surpassing DCGAN (285.0), ACGAN (222.5), WGAN (176.1), and PGGAN (33.76).
• In vessel segmentation on the DRIVE dataset, pretraining with SkrGAN-generated images increased sensitivity from 0.7781 to 0.8464, a relative improvement of 8.78%.
• The AUC and accuracy also improved when using SkrGAN-generated data for pretraining, confirming its effectiveness as a data augmentation strategy.