[Paper Review] Generalized Zero-Shot Learning via VAE-Conditioned Generative Flow

Yuchao Gu, Le Zhang+3 more|arXiv (Cornell University)|Sep 1, 2020

Domain Adaptation and Few-Shot Learning27 references4 citations

TL;DR

This paper proposes VAE-cFlow, a novel conditional generative model for generalized zero-shot learning (GZSL) that combines variational autoencoders (VAE) with normalizing flows to generate high-quality visual features for unseen classes. By encoding semantic descriptions into a meaningful, discriminative latent space and optimizing exact log-likelihood via invertible flows, the method achieves state-of-the-art performance, especially on large-scale datasets like ImageNet21K, with improvements of 3.9% and 3.1% on M500 and M1K splits respectively.

ABSTRACT

Generalized zero-shot learning (GZSL) aims to recognize both seen and unseen classes by transferring knowledge from semantic descriptions to visual representations. Recent generative methods formulate GZSL as a missing data problem, which mainly adopts GANs or VAEs to generate visual features for unseen classes. However, GANs often suffer from instability, and VAEs can only optimize the lower bound on the log-likelihood of observed data. To overcome the above limitations, we resort to generative flows, a family of generative models with the advantage of accurate likelihood estimation. More specifically, we propose a conditional version of generative flows for GZSL, i.e., VAE-Conditioned Generative Flow (VAE-cFlow). By using VAE, the semantic descriptions are firstly encoded into tractable latent distributions, conditioned on that the generative flow optimizes the exact log-likelihood of the observed visual features. We ensure the conditional latent distribution to be both semantic meaningful and inter-class discriminative by i) adopting the VAE reconstruction objective, ii) releasing the zero-mean constraint in VAE posterior regularization, and iii) adding a classification regularization on the latent variables. Our method achieves state-of-the-art GZSL results on five well-known benchmark datasets, especially for the significant improvement in the large-scale setting. Code is released at https://github.com/guyuchao/VAE-cFlow-ZSL.

Motivation & Objective

• To address the limitations of GANs and VAEs in generalized zero-shot learning (GZSL), particularly training instability and suboptimal likelihood estimation.
• To develop a conditional generative model that produces realistic, diverse visual features for unseen classes while preserving semantic meaning.
• To improve GZSL performance by modeling accurate conditional probability distributions of visual features given semantic descriptions.
• To ensure the latent space is both semantically meaningful and inter-class discriminative through joint optimization of reconstruction and classification objectives.
• To demonstrate the importance of matching intra-class variation in synthetic features to real seen-class features for optimal GZSL performance.

Proposed method

• The method uses a VAE to encode semantic descriptions into a tractable, semantic-meaningful latent distribution, replacing the standard zero-mean prior with a classification-regularized posterior.
• A conditional normalizing flow (cFlow) is trained to model the exact log-likelihood of visual features given the VAE-embedded latent codes, enabling precise density estimation.
• The VAE loss ensures that the latent codes can reconstruct the original semantic embeddings, enforcing semantic meaningfulness.
• Posterior regularization is modified by replacing the zero-mean constraint with a classification loss on the latent variables to enhance inter-class discriminability.
• The sampling temperature in the latent space controls intra-class variation, with optimal performance achieved when synthetic feature variation matches that of real seen features.
• The model generates synthetic unseen features at a scale of 12× the number of seen features, with temperature tuned to 1.1 for best performance.

Experimental results

Research questions

• RQ1 Can a conditional normalizing flow be effectively combined with a VAE to generate high-quality visual features for unseen classes in GZSL?
• RQ2 How does the choice of latent space prior and regularization affect the semantic meaningfulness and discriminability of the conditional latent distribution?
• RQ3 What is the impact of intra-class variation in synthetic features on GZSL performance, and how can it be controlled?
• RQ4 Does optimizing the exact log-likelihood via normalizing flows outperform VAEs (which optimize a lower bound) and GANs (which suffer from mode collapse) in GZSL?
• RQ5 How does the proposed method scale across different dataset sizes, particularly in large-scale settings like ImageNet21K?

Key findings

• VAE-cFlow achieves state-of-the-art performance on five benchmark datasets, including a 3.9% improvement on the M500 split and 3.1% on the M1K split of ImageNet21K.
• The method improves GZSL performance by 5.2% and 3.6% on CUB for GZSL and ZSL, respectively, when replacing standard VAE posterior regularization with classification regularization.
• The optimal sampling temperature for synthetic feature generation is 1.1, which matches the intra-class variation of real seen features and maximizes GZSL accuracy.
• Performance improves steadily with increasing numbers of synthetic unseen features, peaking at 12× the number of seen features.
• t-SNE visualization confirms that synthetic features generated by VAE-cFlow are more consistent with real data than those from prior methods.
• The method demonstrates superior performance in large-scale GZSL, though it shows limitations in small-scale ZSL settings due to the nature of VAE-based latent modeling.