[Paper Review] Multi-modal Cycle-consistent Generalized Zero-Shot Learning

Rafael Felix, B. G. Vijay Kumar+2 more|arXiv (Cornell University)|Aug 1, 2018

Domain Adaptation and Few-Shot Learning18 references28 citations

TL;DR

This paper proposes a multi-modal cycle-consistent GAN regularization for generalized zero-shot learning (GZSL) that enforces synthetic visual features to reconstruct their original semantic features, improving generalization to unseen classes. By introducing a cycle consistency loss, the method generates more semantically faithful visual representations, achieving state-of-the-art performance on CUB, FLO, SUN, AWA, and ImageNet datasets.

ABSTRACT

In generalized zero shot learning (GZSL), the set of classes are split into seen and unseen classes, where training relies on the semantic features of the seen and unseen classes and the visual representations of only the seen classes, while testing uses the visual representations of the seen and unseen classes. Current methods address GZSL by learning a transformation from the visual to the semantic space, exploring the assumption that the distribution of classes in the semantic and visual spaces is relatively similar. Such methods tend to transform unseen testing visual representations into one of the seen classes' semantic features instead of the semantic features of the correct unseen class, resulting in low accuracy GZSL classification. Recently, generative adversarial networks (GAN) have been explored to synthesize visual representations of the unseen classes from their semantic features - the synthesized representations of the seen and unseen classes are then used to train the GZSL classifier. This approach has been shown to boost GZSL classification accuracy, however, there is no guarantee that synthetic visual representations can generate back their semantic feature in a multi-modal cycle-consistent manner. This constraint can result in synthetic visual representations that do not represent well their semantic features. In this paper, we propose the use of such constraint based on a new regularization for the GAN training that forces the generated visual features to reconstruct their original semantic features. Once our model is trained with this multi-modal cycle-consistent semantic compatibility, we can then synthesize more representative visual representations for the seen and, more importantly, for the unseen classes. Our proposed approach shows the best GZSL classification results in the field in several publicly available datasets.

Motivation & Objective

• To address the poor generalization of GZSL models to unseen classes due to unconstrained GAN-generated visual features.
• To improve the semantic fidelity of synthesized visual representations for both seen and unseen classes in GZSL.
• To reduce bias toward seen classes by enforcing a cycle-consistent mapping between semantic and visual features.
• To enhance GAN-based GZSL performance through a novel multi-modal cycle consistency regularization.
• To achieve state-of-the-art results across diverse benchmarks including CUB, FLO, SUN, AWA, and ImageNet.

Proposed method

• Proposes a multi-modal cycle consistency loss that enforces the reconstruction of original semantic features from generated visual features.
• Integrates the cycle consistency loss as a regularization term in the GAN training objective to constrain the generator's output.
• Uses a generator network to synthesize visual features from semantic embeddings of both seen and unseen classes.
• Employs a discriminator to distinguish real from generated visual features, ensuring realistic distributional alignment.
• Trains the model end-to-end with a combined loss function combining adversarial loss, classification loss, and cycle consistency loss.
• Applies the trained generator to synthesize visual features for unseen classes, which are then used to train a multi-class classifier.

Experimental results

Research questions

• RQ1 Can enforcing cycle consistency between generated visual features and their source semantic features improve GZSL classification accuracy?
• RQ2 Does the proposed regularization reduce the bias of GZSL models toward seen classes?
• RQ3 How does the cycle-consistent GAN approach compare to state-of-the-art methods like f-CLSWGAN in terms of zero-shot and generalized zero-shot accuracy?
• RQ4 Does the cycle consistency loss lead to faster convergence during training?
• RQ5 How effective is the method on large-scale datasets with high class imbalance and large numbers of classes?

Key findings

• The proposed cycle-consistent GAN approach achieves state-of-the-art performance on CUB, FLO, SUN, AWA, and ImageNet datasets in both ZSL and GZSL settings.
• On CUB, FLO, and AWA, the method significantly outperforms the f-CLSWGAN baseline, with improvements attributed to better semantic fidelity in synthesized features.
• The reconstruction loss ℓREG decreases steadily over training, confirming that the model successfully maps generated visual features back to their original semantic features.
• The cycle-WGAN variant converges faster than the baseline on three out of four datasets, indicating improved training dynamics.
• The cycle-CLSWGAN variant shows comparable convergence speed to the baseline when the classification loss is included, suggesting stable optimization.
• Despite the large class count and high seen/unseen imbalance in the SUN dataset, the cycle-WGAN model still achieves strong performance, though cycle-CLSWGAN performs best there.