[Paper Review] Latent Embedding Feedback and Discriminative Features for Zero-Shot Classification

Sanath Narayan, Akshita Gupta+3 more|arXiv (Cornell University)|Mar 17, 2020

Domain Adaptation and Few-Shot LearningComputer Science48 references33 citations

TL;DR

Proposes TF-VAEGAN, a VAE-GAN framework with a semantic embedding decoder and a feedback loop to enforce semantic consistency across training, synthesis, and classification for (G)ZSL on images and videos, achieving state-of-the-art results on multiple benchmarks.

ABSTRACT

Zero-shot learning strives to classify unseen categories for which no data is available during training. In the generalized variant, the test samples can further belong to seen or unseen categories. The state-of-the-art relies on Generative Adversarial Networks that synthesize unseen class features by leveraging class-specific semantic embeddings. During training, they generate semantically consistent features, but discard this constraint during feature synthesis and classification. We propose to enforce semantic consistency at all stages of (generalized) zero-shot learning: training, feature synthesis and classification. We first introduce a feedback loop, from a semantic embedding decoder, that iteratively refines the generated features during both the training and feature synthesis stages. The synthesized features together with their corresponding latent embeddings from the decoder are then transformed into discriminative features and utilized during classification to reduce ambiguities among categories. Experiments on (generalized) zero-shot object and action classification reveal the benefit of semantic consistency and iterative feedback, outperforming existing methods on six zero-shot learning benchmarks. Source code at https://github.com/akshitac8/tfvaegan.

Motivation & Objective

• Improve zero-shot and generalized zero-shot classification by enforcing semantic consistency at all stages (training, synthesis, and classification).
• Leverage a semantic embedding decoder (SED) to provide feedback and reconstruct semantic embeddings during synthesis and classification.
• Introduce a feedback module to iteratively refine synthesized features within a VAE-GAN framework.
• Enhance discriminative feature transformation during classification by utilizing latent embeddings from the SED.
• Demonstrate improvements on standard ZSL/GZSL benchmarks for images and zero-shot action recognition in videos.

Proposed method

• Build TF-VAEGAN on a f-VAEGAN backbone to synthesize unseen class features using a conditional generator and a VAE encoder.
• Introduce a semantic embedding decoder (SED) that reconstructs embeddings from features and enforces cycle-consistency on embeddings during training (L_R).
• Add a feedback module (F) that modulates generator layers via decoded embeddings to refine feature synthesis (iterative refinement).
• Apply a discriminative feature transformation by concatenating visual features with latent embeddings from the SED for ZSL/GZSL classification (f_zsl and f_gzsl).
• Train with a total loss L_total = L_vaegan + beta * L_R, where L_R enforces embedding-cycle-consistency; extend training to alternating G and F updates.
• Extend framework to generalized zero-shot learning and test on both image datasets (CUB, FLO, SUN, AWA2) and zero-shot action datasets (HMDB51, UCF101).

Experimental results

Research questions

• RQ1 Can semantic consistency be enforced at all stages of (G)ZSL to improve synthesis and discrimination?
• RQ2 Does a semantic embedding decoder provide useful feedback to refine generated features for better ZSL/GZSL performance?
• RQ3 Does incorporating discriminative feature transformation with SED latent embeddings reduce inter-class ambiguity in ZSL/GZSL?
• RQ4 Are these ideas transferable to generalized zero-shot action recognition in videos?
• RQ5 How do the proposed TF-VAEGAN components perform relative to baseline f-VAEGAN and other GAN-based ZSL methods on standard benchmarks?

Key findings

• TF-VAEGAN outperforms baseline f-VAEGAN on four standard ZSL/GZSL datasets in both inductive and transductive settings.
• In ZSL, TF-VAEGAN achieves higher top-1 accuracy across CUB, FLO, SUN, and AWA compared to the baseline and several contemporaries.
• In GZSL, TF-VAEGAN yields higher harmonic means and unseen/seen accuracies across datasets, including transductive settings that leverage unlabelled data.
• Ablation shows that both Feedback and T-feature contributions improve performance beyond the baseline, with TF-VAEGAN combining them for best results.
• TF-VAEGAN also improves over f-CLSWGAN when integrated into that framework, illustrating generalizability of the proposed components.