[Paper Review] Low-shot Learning via Covariance-Preserving Adversarial Augmentation Networks

Hang Gao, Zheng Shou+3 more|arXiv (Cornell University)|Oct 27, 2018

Domain Adaptation and Few-Shot Learning81 citations

TL;DR

The paper introduces Covariance-Preserving Adversarial Augmentation Networks (CP-AAN) to model latent distributions for novel classes using related base classes, preserving covariance to generate diverse, realistic features for low-shot learning, achieving state-of-the-art results on ImageNet.

ABSTRACT

Deep neural networks suffer from over-fitting and catastrophic forgetting when trained with small data. One natural remedy for this problem is data augmentation, which has been recently shown to be effective. However, previous works either assume that intra-class variances can always be generalized to new classes, or employ naive generation methods to hallucinate finite examples without modeling their latent distributions. In this work, we propose Covariance-Preserving Adversarial Augmentation Networks to overcome existing limits of low-shot learning. Specifically, a novel Generative Adversarial Network is designed to model the latent distribution of each novel class given its related base counterparts. Since direct estimation of novel classes can be inductively biased, we explicitly preserve covariance information as the `variability' of base examples during the generation process. Empirical results show that our model can generate realistic yet diverse examples, leading to substantial improvements on the ImageNet benchmark over the state of the art.

Motivation & Objective

• Motivate learning new concepts from very few examples and mitigate catastrophic forgetting during fine-tuning.
• Propose a GAN-based framework that translates base-class features to novel-class features while modeling latent distributions.
• Preserve intra-class covariance to maintain realistic variability when generating novel-class data.
• Introduce Neighborhood Batch Sampling to select related base classes for translation.
• Demonstrate improved low-shot and generalized performance on ImageNet over prior methods.

Proposed method

• Formulate imbalanced set-to-set translation where G_n translates base class features to novel classes.
• Use Neighborhood Batch Sampling (NBS) to pair novel classes with semantically similar base classes based on class prototypes in a learned embedding space.
• Develop several GAN variants: c-GAN, cCyc-GAN, cDeLi-GAN, and the covariance-preserving cCov-GAN.
• Introduce a covariance distance term d_cov that measures discrepancy between base-class and generated-class feature covariances using a truncated SVD-based metric.
• Define L_cov as the expectation over novel-base pairs of the covariance distance, guiding generation to preserve variability.
• Combine adversarial loss, cycle-consistency loss, and L_cov into the final objective for G_n and G_b in cCov-GAN.
• Two-stage training: meta-training with base data to mimic low-shot at test time, then meta-testing with novel classes and their low-shot examples to augment data for final classifier training.
• Train using Prototypical Network embeddings or ResNet embeddings, with episodic sampling and standard GAN optimization.

Experimental results

Research questions

• RQ1 Can covariance-preserving adversarial augmentation produce more diverse and realistic novel-class features compared to naive or cycle-consistent GANs?
• RQ2 Does maintaining base-class intra-class covariance during translation improve low-shot and generalized recognition on ImageNet?
• RQ3 How effective is Neighborhood Batch Sampling in selecting translatable base classes for novel classes?
• RQ4 What is the impact of different GAN variants (c-GAN, cCyc-GAN, cDeLi-GAN, cCov-GAN) on both generation diversity and downstream classification accuracy?

Key findings

• CP-AAN with covariance preservation (cCov-GAN) achieves the best performance among tested variants, showing substantial gains over prior augmentation methods.
• In low-shot learning on ImageNet, cCov-GAN improves top-5 accuracy across K-shot settings, outperforming baselines by notable margins.
• Covariance preservation leads to more diverse and realistic generated features, facilitating better generalization to novel classes.
• Neighborhood Batch Sampling (NBS) improves low-shot recognition compared to standard batch sampling, with soft NBS (NBS-S) generally preferred.
• Ablations show cycle-consistency and Gaussian noise variants improve generation but may misestimate diversity; covariance-based loss yields the most faithful diversity and classification gains.
• Quantitative results indicate the proposed method achieves state-of-the-art performance on ImageNet under both conventional low-shot learning (LSL) and generalized (GLSL) settings.