[Paper Review] LatentAugment: Data Augmentation via Guided Manipulation of GAN's Latent Space

Lorenzo Tronchin, Minh H. Vu+2 more|arXiv (Cornell University)|Jul 21, 2023

Generative Adversarial Networks and Image SynthesisComputer Science3 citations

TL;DR

LatentAugment proposes a GAN-based data augmentation method that enhances diversity and fidelity by guided manipulation of latent vectors in the generator's latent space. It improves generalization in downstream tasks like MRI-to-CT translation, outperforming standard data augmentation and standard GAN sampling in both diversity and mode coverage while maintaining high-speed inference.

ABSTRACT

Data Augmentation (DA) is a technique to increase the quantity and diversity of the training data, and by that alleviate overfitting and improve generalisation. However, standard DA produces synthetic data for augmentation with limited diversity. Generative Adversarial Networks (GANs) may unlock additional information in a dataset by generating synthetic samples having the appearance of real images. However, these models struggle to simultaneously address three key requirements: fidelity and high-quality samples; diversity and mode coverage; and fast sampling. Indeed, GANs generate high-quality samples rapidly, but have poor mode coverage, limiting their adoption in DA applications. We propose LatentAugment, a DA strategy that overcomes the low diversity of GANs, opening up for use in DA applications. Without external supervision, LatentAugment modifies latent vectors and moves them into latent space regions to maximise the synthetic images' diversity and fidelity. It is also agnostic to the dataset and the downstream task. A wide set of experiments shows that LatentAugment improves the generalisation of a deep model translating from MRI-to-CT beating both standard DA as well GAN-based sampling. Moreover, still in comparison with GAN-based sampling, LatentAugment synthetic samples show superior mode coverage and diversity. Code is available at: https://github.com/ltronchin/LatentAugment.

Motivation & Objective

• To address the lack of diversity in GAN-based data augmentation, which limits its use in low-data regimes.
• To improve mode coverage and sample fidelity in GAN-generated images without sacrificing sampling speed.
• To develop a task-agnostic, efficient data augmentation strategy that works across different datasets and downstream tasks.
• To overcome the generative learning trilemma—fidelity, diversity, and speed—by focusing on latent space navigation.
• To enable high-quality, diverse synthetic data generation that better captures the underlying data distribution than standard augmentation or naive GAN sampling.

Proposed method

• The method starts from the latent code of real images and applies gradient-based optimization to steer it toward regions of the latent space that maximize diversity and fidelity.
• It uses perceptual and spatial diversity metrics to guide the optimization, ensuring generated images vary meaningfully while preserving semantic content and style.
• Fidelity is enforced via a pre-trained Inception score or Fréchet Inception Distance (FID) surrogate, and realness is maximized using a discriminator score.
• The approach operates entirely in the latent space, reducing computational cost compared to image-space manipulation.
• It is agnostic to the GAN architecture and downstream task, enabling broad applicability.
• The method does not require external supervision or domain-specific augmentation policies, relying only on latent space optimization.

Figure 1 : Intuition. Considering $\mathcal{X}$ and $\mathcal{W}$ as the data and latent space distributions, respectively. In panel (a), we denote with blue stars the encoded real images. Panel (b). The standard SG2 DA procedure does not provide any guide to the generation process. Thus random late

Experimental results

Research questions

• RQ1 Can guided latent space manipulation in GANs significantly improve the diversity and mode coverage of synthetic data for data augmentation?
• RQ2 Does the proposed method outperform standard data augmentation and standard GAN sampling in terms of downstream model generalization?
• RQ3 Can the method maintain high-fidelity, realistic samples while achieving superior diversity without increasing inference time?
• RQ4 Is the method robust and effective across different datasets and downstream tasks without requiring task-specific tuning?
• RQ5 How does the method compare to diffusion models in terms of generation speed and quality for data augmentation?

Key findings

• LatentAugment improved the generalization performance of a deep learning model in MRI-to-CT translation, outperforming both standard data augmentation and standard GAN-based sampling.
• The method achieved higher precision and recall than standard GAN sampling, indicating superior mode coverage and diversity in generated samples.
• LatentAugment-generated images showed smoother, more realistic variations of real images, preserving main structure and texture while introducing meaningful diversity.
• The method maintained high-speed inference, with a throughput comparable to standard GAN sampling and significantly faster than diffusion models.
• Compared to standard GAN sampling, LatentAugment reduced the generation of out-of-distribution samples, improving data quality and in-distribution coverage.
• The approach is effective without requiring domain knowledge or task-specific tuning, demonstrating broad applicability across datasets and tasks.

Figure 2 : High-level schematic representation of LatentAugment policy (dashed rectangle). It first retrieves the latent code $w^{*}$ of the image $x$ we aim to augment. Then for each iteration $k$ of the procedure, we manipulate $w^{k}$ minimising the loss $\mathcal{L}(w^{k})$ , which is given by a