[Paper Review] Adversarial Training Helps Transfer Learning via Better Representations

Zhun Deng, Linjun Zhang+3 more|arXiv (Cornell University)|Jun 18, 2021

Adversarial Robustness in Machine Learning58 references4 citations

TL;DR

This paper provides a theoretical analysis showing that adversarial training on source data improves transfer learning by learning better representations, leading to more accurate predictors on target tasks. It further demonstrates that semi-supervised learning and adversarial training are complementary, with joint use yielding superior transfer performance on ImageNet and CIFAR benchmarks.

ABSTRACT

Transfer learning aims to leverage models pre-trained on source data to efficiently adapt to target setting, where only limited data are available for model fine-tuning. Recent works empirically demonstrate that adversarial training in the source data can improve the ability of models to transfer to new domains. However, why this happens is not known. In this paper, we provide a theoretical model to rigorously analyze how adversarial training helps transfer learning. We show that adversarial training in the source data generates provably better representations, so fine-tuning on top of this representation leads to a more accurate predictor of the target data. We further demonstrate both theoretically and empirically that semi-supervised learning in the source data can also improve transfer learning by similarly improving the representation. Moreover, performing adversarial training on top of semi-supervised learning can further improve transferability, suggesting that the two approaches have complementary benefits on representations. We support our theories with experiments on popular data sets and deep learning architectures.

Motivation & Objective

• To theoretically explain why adversarial training on source data improves transfer learning performance, despite potential drops in source task accuracy.
• To investigate whether semi-supervised learning via pseudo-labeling also improves representations for transfer learning.
• To examine the complementary benefits of combining adversarial training and pseudo-labeling in the source domain for enhanced transferability.
• To generalize prior transfer learning analysis from regression to classification settings, where adversarial training and pseudo-labeling are more commonly applied.
• To provide a tractable theoretical framework for understanding factors that improve transfer learning in deep learning.

Proposed method

• Theoretical analysis of adversarial training's impact on representation quality using KL divergence and packing numbers to bound generalization error.
• Derivation of representation quality bounds via sinθ distance and log M ≍ rd log(1/η) for η-packing of orthogonal matrices.
• Application of information-theoretic tools (e.g., KL divergence, mutual information) to quantify representation robustness and generalization.
• Empirical validation using ResNet-18 on ImageNet and CIFAR-10/100, with fine-tuning only on the last layer.
• Implementation of adversarial training with ℓ₂ perturbations (ε=1) and pseudo-labeling using a model trained on 10% of ImageNet.
• Systematic ablation of pseudo-label amounts (0% to 900%) to assess their impact on transfer accuracy.

Experimental results

Research questions

• RQ1 Why does adversarial training on the source domain improve transfer learning performance despite possible degradation in source task accuracy?
• RQ2 Can semi-supervised learning via pseudo-labeling similarly improve representations for transfer learning?
• RQ3 Do adversarial training and pseudo-labeling have complementary effects on representation quality in transfer learning?
• RQ4 How does the theoretical analysis of representation quality in classification settings compare to prior regression-based frameworks?
• RQ5 What is the quantitative impact of combining adversarial training and pseudo-labeling on transfer performance across multiple datasets?

Key findings

• Adversarial training on ImageNet improves transfer accuracy on CIFAR-10 by up to 4.3 percentage points (from 17.3% to 21.2%) compared to standard training.
• Pseudo-labeling with 900% of the training set size increased transfer accuracy on the Pets dataset from 43.2% to 59.6% when combined with adversarial training.
• On the Flowers dataset, adversarial training alone increased transfer accuracy from 60.7% to 66.9%, while pseudo-labeling alone improved it to 66.5%.
• The combination of adversarial training and pseudo-labeling yielded the highest performance, with 70.1% accuracy on the Flowers dataset, outperforming either method alone.
• Even when source task accuracy decreased due to adversarial training, transfer performance improved, confirming that representation quality—not source accuracy—drives transfer success.
• Theoretical analysis confirms that adversarial training leads to better representations by reducing generalization error bounds via improved KL divergence control.