[Paper Review] Evolving Semantic Prototype Improves Generative Zero-Shot Learning

Shiming Chen, Wenjin Hou+6 more|arXiv (Cornell University)|Jun 12, 2023

Domain Adaptation and Few-Shot Learning4 citations

TL;DR

This paper proposes Dynamic Semantic Prototype Evolving (DSP), a method that iteratively refines predefined semantic prototypes to reduce visual-semantic domain shift in generative zero-shot learning. By jointly optimizing visual features and semantic prototypes via a V2SM and VOPE network, DSP enables generators to synthesize features closer to real ones, improving state-of-the-art ZSL methods by 8.5%, 8.0%, and 9.7% on CUB, SUN, and AWA2 respectively.

ABSTRACT

In zero-shot learning (ZSL), generative methods synthesize class-related sample features based on predefined semantic prototypes. They advance the ZSL performance by synthesizing unseen class sample features for better training the classifier. We observe that each class's predefined semantic prototype (also referred to as semantic embedding or condition) does not accurately match its real semantic prototype. So the synthesized visual sample features do not faithfully represent the real sample features, limiting the classifier training and existing ZSL performance. In this paper, we formulate this mismatch phenomenon as the visual-semantic domain shift problem. We propose a dynamic semantic prototype evolving (DSP) method to align the empirically predefined semantic prototypes and the real prototypes for class-related feature synthesis. The alignment is learned by refining sample features and semantic prototypes in a unified framework and making the synthesized visual sample features approach real sample features. After alignment, synthesized sample features from unseen classes are closer to the real sample features and benefit DSP to improve existing generative ZSL methods by 8.5\%, 8.0\%, and 9.7\% on the standard CUB, SUN AWA2 datasets, the significant performance improvement indicates that evolving semantic prototype explores a virgin field in ZSL.

Motivation & Objective

• To address the visual-semantic domain shift caused by misaligned predefined semantic prototypes and real class prototypes in generative zero-shot learning.
• To improve the fidelity of synthesized visual features for unseen classes by evolving semantic prototypes toward real prototypes.
• To develop a flexible, end-to-end framework that enhances classifier training through better-aligned generative features.
• To demonstrate that prototype evolution is a promising new direction for advancing generative ZSL methods.

Proposed method

• The method introduces a visual-to-semantic mapping network (V2SM) that infers semantic prototypes from real visual sample features.
• A visual-oriented semantic prototype evolving network (VOPE) iteratively refines the current semantic prototype to better align with real prototypes.
• The generator synthesizes visual features conditioned on the evolving semantic prototype, which are then fed back to V2SM for prototype estimation.
• VOPE is supervised by the V2SM output, enabling joint refinement of prototypes and visual features through a cycle-consistent loss.
• The framework uses a smooth fusion strategy with hyperparameter α to stabilize prototype evolution across steps.
• During inference, evolved prototypes are concatenated with visual features for enhanced semantic conditioning and improved classification.

Experimental results

Research questions

• RQ1 How does the misalignment between predefined semantic prototypes and real prototypes affect generative zero-shot learning performance?
• RQ2 Can iterative refinement of semantic prototypes reduce visual-secular domain shift and improve feature synthesis fidelity?
• RQ3 To what extent can a jointly optimized prototype evolution framework enhance classifier performance across standard ZSL benchmarks?
• RQ4 How robust is the proposed method to hyperparameter variations in prototype evolution and feature synthesis?

Key findings

• The proposed DSP method improves the harmonic mean accuracy of state-of-the-art generative ZSL methods by 8.5% on the CUB dataset.
• On the SUN dataset, DSP achieves an 8.0% improvement in harmonic mean accuracy over baseline generative ZSL methods.
• On the AWA2 dataset, DSP delivers a 9.7% relative improvement in harmonic mean accuracy, demonstrating strong generalization.
• t-SNE visualization confirms that synthesized features from DSP are significantly closer to real features than baseline methods.
• Hyperparameter ablation shows that the method is robust to variations in α, N_syn, and loss weights, with optimal performance at α=0.9, N_syn=800, λ_Scyc=0.1, and λ_V2S=0.6.
• The V2SM supervision on VOPE is critical, as λ_V2S must be larger than other loss weights to ensure effective prototype alignment.