[Paper Review] Zero-shot Learning via Shared-Reconstruction-Graph Pursuit

Bo Zhao, Xinwei Sun+2 more|arXiv (Cornell University)|Nov 20, 2017

Domain Adaptation and Few-Shot Learning18 references9 citations

TL;DR

This paper proposes a novel graph-based method, Shared-Reconstruction-Graph Pursuit (SRG), to address the space shift problem in zero-shot learning (ZSL), where distribution shifts between image and semantic embedding spaces hinder knowledge transfer. By learning shared reconstruction coefficients across image and semantic spaces, SRG enables synthesis of unseen class prototypes for classification, achieving state-of-the-art performance on three benchmark datasets, including 18.26% Top-5 accuracy on ImageNet.

ABSTRACT

Zero-shot learning (ZSL) aims to recognize objects from novel unseen classes without any training data. Recently, structure-transfer based methods are proposed to implement ZSL by transferring structural knowledge from the semantic embedding space to image feature space to classify testing images. However, we observe that such a knowledge transfer framework may suffer from the problem of the geometric inconsistency between the data in the training and testing spaces. We call this problem as the space shift problem. In this paper, we propose a novel graph based method to alleviate this space shift problem. Specifically, a Shared Reconstruction Graph (SRG) is pursued to capture the common structure of data in the two spaces. With the learned SRG, each unseen class prototype (cluster center) in the image feature space can be synthesized by the linear combination of other class prototypes, so that testing instances can be classified based on the distance to these synthesized prototypes. The SRG bridges the image feature space and semantic embedding space. By applying spectral clustering on the learned SRG, many meaningful clusters can be discovered, which interprets ZSL performance on the datasets. Our method can be easily extended to the generalized zero-shot learning setting. Experiments on three popular datasets show that our method outperforms other methods on all datasets. Even with a small number of training samples, our method can achieve the state-of-the-art performance.

Motivation & Objective

• Address the space shift problem in zero-shot learning, where training and testing data reside in geometrically inconsistent image and semantic embedding spaces.
• Develop a method to transfer structural knowledge from semantic space to image feature space without relying on direct mapping functions.
• Enable accurate classification of unseen classes by synthesizing their prototypes using shared reconstruction coefficients.
• Improve generalization in generalized zero-shot learning (GZSL) by discovering meaningful latent clusters through spectral clustering on the SRG.
• Enhance robustness with sparse and locally structured reconstruction, especially effective with limited training samples.

Proposed method

• Construct a Shared Reconstruction Graph (SRG) by learning shared reconstruction coefficients that map each class prototype in the image space to a linear combination of other prototypes in both image and semantic embedding spaces.
• Formulate the SRG learning as an optimization problem with sparsity and locality regularization to select relevant, informative prototypes for reconstruction.
• Apply spectral clustering on the learned SRG to discover meaningful, interpretable clusters in the embedding space, which aids in knowledge transfer.
• Synthesize unseen class prototypes in the image space using the shared reconstruction coefficients derived from the semantic space, enabling classification via nearest-neighbor distance.
• Extend the method to generalized ZSL by evaluating performance on both seen and unseen classes, using metrics like ${u\rightarrow u}$, ${s\rightarrow s}$, and ${s\rightarrow\tau}$.
• Utilize both attributes and word vectors as semantic embeddings, with fusion via regularization to improve robustness and performance.

Experimental results

Research questions

• RQ1 How can structural knowledge be effectively transferred between image and semantic embedding spaces in zero-shot learning despite geometric inconsistencies?
• RQ2 To what extent does the proposed Shared-Reconstruction-Graph (SRG) mitigate the space shift problem in ZSL?
• RQ3 Can sparse and locally structured reconstruction improve prototype synthesis and classification accuracy, especially with few training samples?
• RQ4 Does the SRG framework enable the discovery of semantically meaningful latent clusters that support effective knowledge transfer?
• RQ5 How does the method perform in the more realistic generalized zero-shot learning (GZSL) setting compared to state-of-the-art approaches?

Key findings

• The proposed SRG method achieves state-of-the-art performance on three benchmark datasets: 18.26% Top-5 accuracy on ImageNet, outperforming DeViSE and ConSE.
• On ImageNet, the method achieves 8.14% Top-1 accuracy, a 0.42% improvement over the previous best (7.72%) when sparsity and locality regularization are applied.
• On AwA, the method achieves the highest accuracy across all metrics in the GZSL setting, with 100% ${u\rightarrow u}$ and 85.7% ${s\rightarrow s}$ accuracy.
• The introduction of sparsity regularization improves performance by up to 16.29% on AwA when using attributes as semantic embeddings, demonstrating its effectiveness.
• Spectral clustering on the SRG reveals 11 meaningful, separable clusters on AwA, where unseen classes are grouped with semantically similar seen classes, validating the interpretability of the method.
• The method maintains strong performance even with a small number of training samples, indicating robustness and generalization capability.