[Paper Review] Synthesized Classifiers for Zero-Shot Learning

Soravit Changpinyo, Wei‐Lun Chao+2 more|arXiv (Cornell University)|Mar 2, 2016

Domain Adaptation and Few-Shot Learning31 references57 citations

TL;DR

This paper proposes a manifold learning-based approach for zero-shot learning that aligns semantic and model spaces using adaptable 'phantom' classes as shared bases. By optimizing these phantom classes to enable convex combination of real classifiers, the method achieves state-of-the-art accuracy on four benchmark datasets, including ImageNet with over 20,000 unseen classes.

ABSTRACT

Given semantic descriptions of object classes, zero-shot learning aims to accurately recognize objects of the unseen classes, from which no examples are available at the training stage, by associating them to the seen classes, from which labeled examples are provided. We propose to tackle this problem from the perspective of manifold learning. Our main idea is to align the semantic space that is derived from external information to the model space that concerns itself with recognizing visual features. To this end, we introduce a set of "phantom" object classes whose coordinates live in both the semantic space and the model space. Serving as bases in a dictionary, they can be optimized from labeled data such that the synthesized real object classifiers achieve optimal discriminative performance. We demonstrate superior accuracy of our approach over the state of the art on four benchmark datasets for zero-shot learning, including the full ImageNet Fall 2011 dataset with more than 20,000 unseen classes.

Motivation & Objective

• Address the challenge of recognizing unseen object classes without labeled training examples.
• Overcome the limitation of existing methods that fail to align semantic embeddings with visual model spaces effectively.
• Improve zero-shot recognition performance by learning a shared representation between semantic and visual model spaces.
• Enable generalization to large-scale datasets with tens of thousands of unseen classes, such as ImageNet.
• Develop a method that synthesizes real classifiers from optimized phantom bases to enhance discriminative performance.

Proposed method

• Introduce 'phantom' object classes whose semantic and model space coordinates are jointly optimized.
• Model both semantic and visual model spaces as weighted graphs, where class relatedness is encoded in edge weights.
• Use manifold learning (e.g., Laplacian eigenmaps) to project semantic space vertices into the model space, preserving class relationships.
• Represent real object classifiers as convex combinations of phantom class classifiers, enabling synthesis of unseen class models.
• Optimize the phantom class coordinates using labeled seen-class data to maximize discriminative accuracy on unseen classes.
• Leverage deep features for better semantic alignment and improved performance over shallow features.

Experimental results

Research questions

• RQ1 How can semantic and visual model spaces be effectively aligned to improve zero-shot generalization?
• RQ2 Can phantom classes serve as shared bases to synthesize high-performing classifiers for unseen classes?
• RQ3 What is the impact of using deep versus shallow features on classifier synthesis performance?
• RQ4 How many phantom (base) classifiers are sufficient to achieve strong performance, especially on fine-grained datasets?
• RQ5 Why do some unseen class images fail to be classified correctly despite semantic similarity to seen classes?

Key findings

• The proposed method achieves state-of-the-art zero-shot recognition accuracy on four benchmark datasets, including the full ImageNet Fall 2011 with over 20,000 unseen classes.
• On the CUB dataset, the method achieves superior performance even when using only 60% of the number of seen classes as phantom bases, indicating high data efficiency.
• The use of deep features significantly outperforms shallow features, attributed to better semantic alignment and lower dimensionality.
• Failure cases are primarily due to visual dissimilarity between test images and the visual characteristics of semantically similar seen classes, despite semantic similarity.
• The method demonstrates robustness on fine-grained recognition tasks, where high class correlation allows effective classifier synthesis with fewer phantom bases.
• PCA analysis shows that CUB requires fewer principal components to capture classifier variance than AwA, explaining better performance with fewer bases on CUB.