[Paper Review] Leveraging the Invariant Side of Generative Zero-Shot Learning

Jingjing Li, Mengmeng Jin+4 more|arXiv (Cornell University)|Apr 8, 2019

Domain Adaptation and Few-Shot LearningComputer Science37 references103 citations

TL;DR

LisGAN directly generates unseen visual features using a conditional Wasserstein GAN guided by semantic descriptions and invariant soul samples, then uses a cascaded classifier for zero-shot recognition, achieving state-of-the-art results on multiple benchmarks.

ABSTRACT

Conventional zero-shot learning (ZSL) methods generally learn an embedding, e.g., visual-semantic mapping, to handle the unseen visual samples via an indirect manner. In this paper, we take the advantage of generative adversarial networks (GANs) and propose a novel method, named leveraging invariant side GAN (LisGAN), which can directly generate the unseen features from random noises which are conditioned by the semantic descriptions. Specifically, we train a conditional Wasserstein GANs in which the generator synthesizes fake unseen features from noises and the discriminator distinguishes the fake from real via a minimax game. Considering that one semantic description can correspond to various synthesized visual samples, and the semantic description, figuratively, is the soul of the generated features, we introduce soul samples as the invariant side of generative zero-shot learning in this paper. A soul sample is the meta-representation of one class. It visualizes the most semantically-meaningful aspects of each sample in the same category. We regularize that each generated sample (the varying side of generative ZSL) should be close to at least one soul sample (the invariant side) which has the same class label with it. At the zero-shot recognition stage, we propose to use two classifiers, which are deployed in a cascade way, to achieve a coarse-to-fine result. Experiments on five popular benchmarks verify that our proposed approach can outperform state-of-the-art methods with significant improvements.

Motivation & Objective

• Motivate zero-shot learning to recognize unseen classes using semantic descriptions without real unseen samples.
• Develop a generative framework that ensures both diversity and reliability of synthesized unseen features.
• Introduce soul samples as invariant representations to regularize generated features.
• Address multi-view domain shifts with multiple soul samples per class.
• Enhance recognition with a coarse-to-fine cascade classifier on generated features.

Proposed method

• Train a conditional Wasserstein GAN to synthesize unseen features from random noise conditioned on semantic descriptions.
• Introduce soul samples as invariant class representations, with multiple soul samples per class to capture multi-view characteristics.
• Define two regularizers (L_R1 and L_R2) to ensure generated samples and soul samples align with real class representations.
• Use a two-branch GAN objective: Wasserstein loss plus supervised classification loss on both real and generated features.
• Convert zero-shot recognition into supervised learning on synthesized features, with a cascade classifier using entropy-based confidence to refine predictions.
• Optimize with Lipschitz constraint (β fixed at 10) and tune λ and regularization weights to balance diversity and alignment.

Experimental results

Research questions

• RQ1 Can conditional GANs generate diverse and discriminative unseen features that align with semantic descriptions?
• RQ2 Do soul samples effectively regularize generation to prevent soulless features and mitigate multi-view domain shift?
• RQ3 Does a cascade classifier leveraging high-confidence unseen samples improve generalized zero-shot performance?
• RQ4 What is the sensitivity of LisGAN to hyperparameters and how does its stability behave during training?

Key findings

• LisGAN achieves best results on four of five zero-shot learning evaluations and state-of-the-art on the remaining dataset.
• On aPaY, LisGAN improves over the previous state-of-the-art by 2.6%.
• On AwA, CUB, and FLO, LisGAN improves by 2.4%, 1.5%, and 2.4% respectively in zero-shot accuracy.
• In generalized zero-shot learning, LisGAN shows harmonic-mean improvements up to 2.8% across datasets, with an average improvement around 2.2%.
• Ablation studies show that soul-sample regularization, multiple soul samples per class, and the cascade classifier jointly contribute to performance gains.