[Paper Review] Zero-Shot Learning via Semantic Similarity Embedding

Ziming Zhang, Venkatesh Saligrama|arXiv (Cornell University)|Sep 15, 2015

Domain Adaptation and Few-Shot LearningComputer Science32 references117 citations

TL;DR

This paper proposes a semantic similarity embedding (SSE) method for zero-shot learning that models source and target domain data as mixtures of seen class proportions, learning class-dependent feature transforms to map both domains into a shared semantic space where similarity measures enable accurate classification of unseen classes. The approach achieves state-of-the-art performance on multiple benchmark datasets, including SUN Attributes, with significant accuracy gains over prior methods.

ABSTRACT

In this paper we consider a version of the zero-shot learning problem where seen class source and target domain data are provided. The goal during test-time is to accurately predict the class label of an unseen target domain instance based on revealed source domain side information (\eg attributes) for unseen classes. Our method is based on viewing each source or target data as a mixture of seen class proportions and we postulate that the mixture patterns have to be similar if the two instances belong to the same unseen class. This perspective leads us to learning source/target embedding functions that map an arbitrary source/target domain data into a same semantic space where similarity can be readily measured. We develop a max-margin framework to learn these similarity functions and jointly optimize parameters by means of cross validation. Our test results are compelling, leading to significant improvement in terms of accuracy on most benchmark datasets for zero-shot recognition.

Motivation & Objective

• To address the challenge of classifying unseen classes in zero-shot learning when only source domain attributes (e.g., attributes, words) are available at test time.
• To improve zero-shot recognition by modeling both source and target domain data as probabilistic mixtures of seen class proportions.
• To learn shared semantic embeddings for source and target domains such that similar mixture patterns indicate the same unseen class.
• To jointly optimize embedding parameters using cross-validation on held-out seen classes for better generalization to unseen classes.
• To overcome limitations of existing methods that rely on noisy side information or linear mappings by introducing class-dependent feature transforms.

Proposed method

• Represents each source domain class as a histogram of seen class proportions, forming a probabilistic mixture on the simplex.
• Projects target domain instances into the same semantic space using class-dependent feature transforms to estimate their seen-class mixture proportions.
• Learns source and target domain embedding functions via a max-margin framework that aligns seen class data distributions across domains.
• Uses two variants: one based on the intersection function and another using the rectified linear unit (ReLU) for similarity scoring.
• Employs cross-validation by holding out subsets of seen classes to jointly optimize embedding parameters for improved generalization.
• Applies deep features (e.g., VGG-verydeep-19) as input to the embedding network, enabling robust feature representation.

Experimental results

Research questions

• RQ1 Can modeling source and target domain data as mixtures of seen class proportions improve zero-shot recognition accuracy?
• RQ2 Does learning class-dependent feature transforms lead to better alignment of semantic affinities between source and target domains?
• RQ3 Can a max-margin framework with cross-validation on held-out seen classes generalize effectively to unseen classes?
• RQ4 How does the proposed method compare to state-of-the-art approaches on large-scale zero-shot recognition benchmarks?
• RQ5 To what extent does the method remain robust when the number of seen classes is small and the number of unseen classes is large?

Key findings

• On the SUN Attributes dataset, the method achieves 87.17% accuracy with 317 seen classes and 10 unseen classes, a significant improvement over random chance (0.14%).
• With 700 unseen classes, performance drops to 2.85% accuracy, but remains stable and significantly above random baseline, indicating scalability.
• On CIFAR-10, the method completes inference in under 5 minutes using a multi-thread CPU, demonstrating computational efficiency.
• The method outperforms existing state-of-the-art approaches on five benchmark datasets, including large-scale zero-shot recognition tasks.
• The use of class-dependent feature transforms and semantic similarity embedding leads to better alignment of target domain data with source domain attributes, especially when using VGG-verydeep-19 features.
• The method generalizes well even with few seen classes, achieving 61.00% accuracy with 17 seen classes and 10 unseen classes, improving to 87.17% with 317 seen classes.