[Paper Review] Zero-Shot Learning via Class-Conditioned Deep Generative Models

Wenlin Wang, Yunchen Pu+6 more|arXiv (Cornell University)|Nov 15, 2017

Domain Adaptation and Few-Shot LearningComputer Science42 references51 citations

TL;DR

Introduces a deep generative model for zero-shot learning where each class is represented by a class-conditioned latent Gaussian prior for a supervised VAE, enabling inductive, transductive, and few-shot ZSL by matching test latent posteriors to unseen class priors.

ABSTRACT

We present a deep generative model for learning to predict classes not seen at training time. Unlike most existing methods for this problem, that represent each class as a point (via a semantic embedding), we represent each seen/unseen class using a class-specific latent-space distribution, conditioned on class attributes. We use these latent-space distributions as a prior for a supervised variational autoencoder (VAE), which also facilitates learning highly discriminative feature representations for the inputs. The entire framework is learned end-to-end using only the seen-class training data. The model infers corresponding attributes of a test image by maximizing the VAE lower bound; the inferred attributes may be linked to labels not seen when training. We further extend our model to a (1) semi-supervised/transductive setting by leveraging unlabeled unseen-class data via an unsupervised learning module, and (2) few-shot learning where we also have a small number of labeled inputs from the unseen classes. We compare our model with several state-of-the-art methods through a comprehensive set of experiments on a variety of benchmark data sets.

Motivation & Objective

• Motivate zero-shot learning when labeled data for unseen classes is unavailable or scarce.
• Represent each class as a latent-space distribution conditioned on class attributes to capture intra-class variability.
• Develop a supervised VAE framework that uses class-conditioned priors to enable recognition of unseen classes.
• Extend the model to semi-supervised/transductive and few-shot settings by leveraging unlabeled data and a small number of labeled unseen-class examples.
• Train end-to-end using only seen-class data (and optional unlabeled data) to learn discriminative latent representations.

Proposed method

• Define a class-conditioned latent Gaussian prior p_ψ(z|a) with μ(a)=W_μ a and Σ(a)=diag(exp(W_σ a)).
• Use a variational lower bound with q_φ(z|x) to maximize E_{q}[log p_θ(x|z)] - KL(q_φ(z|x)||p_ψ(z|a)).
• Incorporate a margin regularizer that encourages q_φ(z|x) to be close to the true class prior p_ψ(z|a) but far from other class priors, approximated by a softmax-based surrogate.
• Extend to transductive ZSL by adding an unlabeled-data regularizer that sharpens unseen-class predictions via KL divergences between q and sharpened class-distribution expectations.
• Enable few-shot learning by incorporating labeled examples from unseen classes and optimizing the same objective over these additional samples.
• Experimentally evaluate on AwA, CUB-200, SUN, and ImageNet using VGG-19 fc7 features and, where applicable, word2vec embeddings for ImageNet.

Experimental results

Research questions

• RQ1 Can class-conditioned latent distributions in a VAE capture intra-class variability and enable accurate ZSL for unseen classes?
• RQ2 Does incorporating unlabeled data (transductive setting) improve zero-shot and few-shot performance relative to purely inductive approaches?
• RQ3 How does a margin-based regularizer influence class discrimination in the latent space and final predictions?
• RQ4 What is the impact of reconstruction terms in the VAE objective on ZSL performance across datasets?
• RQ5 Can the method scale to large datasets (e.g., ImageNet) and leverage different semantic representations (attributes vs. word vectors)?

Key findings

• The proposed VZSL method outperforms state-of-the-art baselines on small-scale datasets (AwA, CUB-200, SUN) and large-scale ImageNet in inductive ZSL.
• In the transductive setting, VZSL yields substantial gains over inductive results, with about 8% average improvement across datasets.
• Model variants with the VAE reconstruction term generally perform better, indicating the benefit of a generative component; the margin regularizer further enhances discrimination when unlabeled data is used.
• Few-shot extensions show that adding a small number of labeled unseen-class examples significantly improves over standard ZSL and can surpass multiclass SVM baselines in transfer learning scenarios.
• t-SNE visualizations illustrate that the latent space learned by VZSL yields more separable class representations compared to raw CNN features and reconstructed features.
• Across evaluations, VZSL demonstrates robustness to domain shift by leveraging unlabeled data from unseen classes.