[Paper Review] Semantic Autoencoder for Zero-Shot Learning

Elyor Kodirov, Tao Xiang+1 more|arXiv (Cornell University)|Apr 26, 2017

Domain Adaptation and Few-Shot LearningComputer Science53 references119 citations

TL;DR

Introduces Semantic AutoEncoder (SAE) that jointly learns an encoder to map visual features to a semantic space and a decoder to reconstruct the original features, improving zero-shot learning generalization and enabling efficient large-scale training.

ABSTRACT

Existing zero-shot learning (ZSL) models typically learn a projection function from a feature space to a semantic embedding space (e.g.~attribute space). However, such a projection function is only concerned with predicting the training seen class semantic representation (e.g.~attribute prediction) or classification. When applied to test data, which in the context of ZSL contains different (unseen) classes without training data, a ZSL model typically suffers from the project domain shift problem. In this work, we present a novel solution to ZSL based on learning a Semantic AutoEncoder (SAE). Taking the encoder-decoder paradigm, an encoder aims to project a visual feature vector into the semantic space as in the existing ZSL models. However, the decoder exerts an additional constraint, that is, the projection/code must be able to reconstruct the original visual feature. We show that with this additional reconstruction constraint, the learned projection function from the seen classes is able to generalise better to the new unseen classes. Importantly, the encoder and decoder are linear and symmetric which enable us to develop an extremely efficient learning algorithm. Extensive experiments on six benchmark datasets demonstrate that the proposed SAE outperforms significantly the existing ZSL models with the additional benefit of lower computational cost. Furthermore, when the SAE is applied to supervised clustering problem, it also beats the state-of-the-art.

Motivation & Objective

• Motivate and address projection domain shift in zero-shot learning (ZSL).
• Propose a semantically guided autoencoder that enforces both encoding and reconstruction constraints.
• Develop an efficient solver for the resulting Sylvester equation independent of training data size.
• Demonstrate state-of-the-art ZSL performance on six benchmarks and show applicability to supervised clustering.

Proposed method

• Model the encoder W and decoder W^T as symmetric, linear projections with a shared latent semantic space.
• Impose WX = S as a hard constraint and relax to a soft constraint with objective min_W ||X - W^T S||_F^2 + λ||WX - S||_F^2.
• Derive a Sylvester equation AW + WB = C with A = SS^T, B = λXX^T, C = (1+λ) SX^T to solve for W (complexity independent of N).
• Provide a 6-line MATLAB implementation of SAE solving the Sylvester equation.
• Apply SAE to zero-shot learning by projecting test visuals into the semantic space and performing nearest-neighbor against unseen class prototypes, or projecting prototypes into the visual space and classifying in that space.
• Show applicability to supervised clustering by learning a semantic projection from labeled data and clustering projected test data.

Experimental results

Research questions

• RQ1 Can a reconstruction constraint in a zero-shot learning encoder improve generalization to unseen classes?
• RQ2 Does a symmetric, linear SAE offer competitive performance with reduced computational cost compared to existing ZSL models?
• RQ3 How does SAE perform on both small-scale (attributes) and large-scale (word vectors) semantic spaces?
• RQ4 Can the SAE framework be effectively extended to supervised clustering beyond ZSL?

Key findings

• SAE achieves state-of-the-art or competitive results on six ZSL benchmarks, outperforming many existing models.
• Using the reconstruction constraint significantly boosts performance over simple projection baselines (ridge regression) in both encoder and decoder settings.
• On large-scale ImNet-2, SAE (W) improves over the state-of-the-art SS-Voc by 8.8%.
• Both SAE variants (encoder W and decoder W^T) are effective for ZSL, with encoder often slightly better overall.
• SAE reduces computational cost: training is at least 10x faster than comparators (e.g., on AwA).
• SAE also benefits supervised clustering, often outperforming dedicated metric-learning methods on synthetic and real data.