[Paper Review] Zero-Shot Learning by Harnessing Adversarial Samples

Zhi Chen, Pengfei Zhang+3 more|arXiv (Cornell University)|Aug 1, 2023

Domain Adaptation and Few-Shot LearningComputer Science3 citations

TL;DR

This paper proposes Harnessing Adversarial Samples (HAS), a novel zero-shot learning (ZSL) method that generates semantic-preserving adversarial augmentations to improve generalization without causing semantic distortion. By enforcing attribute-conditioned, latent-space-constrained, and diverse perturbations, HAS achieves state-of-the-art performance on CUB, FGVCAircraft, and Stanford Cars datasets in both ZSL and generalized ZSL settings.

ABSTRACT

Zero-Shot Learning (ZSL) aims to recognize unseen classes by generalizing the knowledge, i.e., visual and semantic relationships, obtained from seen classes, where image augmentation techniques are commonly applied to improve the generalization ability of a model. However, this approach can also cause adverse effects on ZSL since the conventional augmentation techniques that solely depend on single-label supervision is not able to maintain semantic information and result in the semantic distortion issue consequently. In other words, image argumentation may falsify the semantic (e.g., attribute) information of an image. To take the advantage of image augmentations while mitigating the semantic distortion issue, we propose a novel ZSL approach by Harnessing Adversarial Samples (HAS). HAS advances ZSL through adversarial training which takes into account three crucial aspects: (1) robust generation by enforcing augmentations to be similar to negative classes, while maintaining correct labels, (2) reliable generation by introducing a latent space constraint to avert significant deviations from the original data manifold, and (3) diverse generation by incorporating attribute-based perturbation by adjusting images according to each semantic attribute's localization. Through comprehensive experiments on three prominent zero-shot benchmark datasets, we demonstrate the effectiveness of our adversarial samples approach in both ZSL and Generalized Zero-Shot Learning (GZSL) scenarios. Our source code is available at https://github.com/uqzhichen/HASZSL.

Motivation & Objective

• To address the semantic distortion problem in zero-shot learning caused by conventional image augmentation techniques that alter pixel values without preserving attribute-level semantics.
• To improve model generalization in ZSL by leveraging adversarial training that generates diverse, robust, and semantically coherent data augmentations.
• To maintain semantic coherence during data augmentation by conditioning perturbations on localized attributes and constraining them within the data manifold.
• To enable more accurate attribute localization and prediction by focusing perturbations on regions most relevant to semantic attributes.
• To demonstrate the superiority of adversarial samples over traditional augmentation in both standard and generalized zero-shot learning scenarios.

Proposed method

• Adversarial training is employed to generate perturbations that are semantically aligned with attribute supervision, ensuring that changes to images do not distort semantic meaning.
• A latent space constraint is applied to keep adversarial samples close to the original data manifold, preventing large deviations that could disrupt semantic learning.
• Attribute-based perturbations are introduced by conditioning the adversarial noise on the spatial localization of specific semantic attributes, enhancing relevance and diversity.
• The method separates foreground and background perturbations, allowing targeted manipulation of object regions while minimizing interference with background context.
• The model is trained to maintain correct attribute predictions under adversarial perturbations, ensuring robustness and semantic fidelity.
• The approach integrates attribute-conditional generation with adversarial loss to produce augmentations that are both diverse and faithful to semantic supervision.

Experimental results

Research questions

• RQ1 Can adversarial samples be effectively harnessed to improve zero-shot learning generalization while preserving semantic integrity?
• RQ2 How can adversarial training be adapted to maintain attribute-level semantics during data augmentation in ZSL?
• RQ3 Does adversarial data generation outperform conventional image augmentation techniques in reducing semantic distortion in ZSL?
• RQ4 To what extent can adversarial perturbations improve attribute localization and prediction accuracy in zero-shot learning?
• RQ5 Can the proposed method generalize across different ZSL benchmarks and settings, including generalized zero-shot learning?

Key findings

• The proposed HAS method achieves state-of-the-art performance on three major zero-shot learning benchmarks: CUB, FGVCAircraft, and Stanford Cars.
• On the CUB dataset, HAS achieves a ZSL accuracy of 58.7% and a GZSL accuracy of 48.9%, outperforming prior methods in both settings.
• The ablation study confirms that all three components—robust, reliable, and diverse generation—contribute significantly to performance gains.
• Visualization of attention maps shows that the model focuses on correct attribute-relevant regions, confirming improved localization due to localized perturbations.
• Perturbation analysis reveals that adversarial noise is predominantly applied to foreground objects, with minimal impact on background, preserving semantic context.
• The method effectively mitigates semantic distortion: unlike traditional augmentation, it avoids incorrect attribute predictions, maintaining clear separation in the attribute space.