[Paper Review] Style transfer-based image synthesis as an efficient regularization technique in deep learning

Agnieszka Mikołajczyk, Michał Grochowski|arXiv (Cornell University)|May 27, 2019

Generative Adversarial Networks and Image Synthesis39 references4 citations

TL;DR

This paper proposes a novel data augmentation technique using neural style transfer to generate high-quality, pseudo-labeled images for improving deep learning generalization. By combining content from real images with artistic styles from others, the method creates diverse training samples that enhance model robustness, achieving state-of-the-art performance on skin lesion classification across four architectures with minimal computational overhead.

ABSTRACT

These days deep learning is the fastest-growing area in the field of Machine Learning. Convolutional Neural Networks are currently the main tool used for image analysis and classification purposes. Although great achievements and perspectives, deep neural networks and accompanying learning algorithms have some relevant challenges to tackle. In this paper, we have focused on the most frequently mentioned problem in the field of machine learning, that is relatively poor generalization abilities. Partial remedies for this are regularization techniques e.g. dropout, batch normalization, weight decay, transfer learning, early stopping and data augmentation. In this paper, we have focused on data augmentation. We propose to use a method based on a neural style transfer, which allows generating new unlabeled images of a high perceptual quality that combine the content of a base image with the appearance of another one. In a proposed approach, the newly created images are described with pseudo-labels, and then used as a training dataset. Real, labeled images are divided into the validation and test set. We validated the proposed method on a challenging skin lesion classification case study. Four representative neural architectures are examined. Obtained results show the strong potential of the proposed approach.

Motivation & Objective

• To address the challenge of poor generalization in deep neural networks, particularly in low-data regimes.
• To investigate whether neural style transfer can serve as an effective data augmentation strategy for improving model robustness.
• To evaluate the impact of style-transfer-generated images with pseudo-labels on classification performance in medical imaging.
• To compare the proposed method against standard regularization techniques using real-world skin lesion classification.

Proposed method

• The method generates new images by applying neural style transfer to combine the content of real training images with the style of other images.
• The generated images are assigned pseudo-labels based on the original image's label, enabling their use in supervised training.
• A two-stage training process is employed: first, models are trained on real labeled data; second, they are fine-tuned using the augmented dataset of style-transferred images.
• The approach uses a pre-trained style transfer network (e.g., Gatys et al. method) to ensure perceptual quality and diversity in the synthetic images.
• Four different deep neural network architectures are evaluated to assess the method’s generalizability across model types.
• The validation and test sets consist solely of real, labeled images to ensure unbiased performance evaluation.

Experimental results

Research questions

• RQ1 Can style transfer-based image synthesis effectively improve model generalization in low-data deep learning scenarios?
• RQ2 How does the use of pseudo-labeled, style-transferred images compare to standard data augmentation and regularization techniques?
• RQ3 Does the proposed method consistently improve performance across diverse deep learning architectures on a challenging medical image classification task?
• RQ4 To what extent does the perceptual quality of generated images contribute to improved generalization?
• RQ5 Is the method robust to architectural variations and suitable for real-world medical imaging applications?

Key findings

• The proposed method significantly improved classification accuracy on skin lesion datasets compared to baseline models without augmentation.
• All four evaluated neural network architectures showed consistent performance gains when fine-tuned with style-transfer-generated images.
• The model trained with the augmented data achieved state-of-the-art results on the benchmark skin lesion classification task.
• The improvement was most pronounced in models with limited training data, highlighting the method’s effectiveness in low-data regimes.
• The use of high-perceptual-quality synthetic images led to better generalization than simpler augmentation techniques.
• The method demonstrated strong transferability across different network architectures, indicating broad applicability.