[Paper Review] Zero-Shot Learning by Convex Combination of Semantic Embeddings

Mohammad Norouzi, Tomáš Mikolov+6 more|arXiv (Cornell University)|Dec 19, 2013

Domain Adaptation and Few-Shot Learning17 references781 citations

TL;DR

This paper proposes Convex Combination of Semantic Embeddings (ConSE), a simple yet effective method that maps images into a semantic embedding space by convexly combining pre-trained word embeddings using the softmax probabilities from any existing n-way image classifier. The approach achieves state-of-the-art zero-shot learning performance on ImageNet, attaining 9.4% hit@1 and 24.7% hit@5 on 1,600 unseen object categories without additional training.

ABSTRACT

Several recent publications have proposed methods for mapping images into continuous semantic embedding spaces. In some cases the embedding space is trained jointly with the image transformation. In other cases the semantic embedding space is established by an independent natural language processing task, and then the image transformation into that space is learned in a second stage. Proponents of these image embedding systems have stressed their advantages over the traditional way{} classification framing of image understanding, particularly in terms of the promise for zero-shot learning -- the ability to correctly annotate images of previously unseen object categories. In this paper, we propose a simple method for constructing an image embedding system from any existing way{} image classifier and a semantic word embedding model, which contains the $ $ class labels in its vocabulary. Our method maps images into the semantic embedding space via convex combination of the class label embedding vectors, and requires no additional training. We show that this simple and direct method confers many of the advantages associated with more complex image embedding schemes, and indeed outperforms state of the art methods on the ImageNet zero-shot learning task.

Motivation & Objective

• To enable zero-shot learning by transforming any pre-trained n-way image classifier into a semantic embedding model without retraining.
• To leverage pre-existing word embeddings and image classifier outputs to construct continuous image representations in a semantic space.
• To evaluate whether a simple, direct method can outperform more complex joint training approaches in zero-shot image recognition.
• To explore the role of confidence and semantic proximity in embedding magnitude and generalization.

Proposed method

• The method uses the softmax probability output of a pre-trained image classifier as weights for a convex combination of corresponding class label word embeddings.
• Each image is embedded as a weighted sum of the semantic vectors of the n class labels, where weights are the predicted class probabilities.
• For multi-synonym labels (e.g., ImageNet synsets), word vectors for all synonyms are averaged before combining with classifier scores.
• During inference, top-k predicted labels are retrieved by ranking all individual word vectors based on their cosine similarity to the image embedding.
• The approach is compatible with any image classifier that outputs relative class scores and any semantic word embedding model where semantically similar concepts are close in vector space.
• The method naturally encodes model confidence through the L2 norm of the resulting embedding, which is lower for ambiguous or semantically diverse predictions.

Experimental results

Research questions

• RQ1 Can a simple convex combination of word embeddings using classifier scores enable effective zero-shot learning?
• RQ2 Does this method outperform more complex joint training approaches in zero-shot image recognition?
• RQ3 How does the model’s confidence, encoded in embedding magnitude, correlate with prediction ambiguity or semantic diversity?
• RQ4 To what extent does the choice of word embedding corpus affect performance on zero-shot generalization?

Key findings

• ConSE achieves 9.4% hit@1 and 24.7% hit@5 on 1,600 unseen ImageNet categories, outperforming a recent state-of-the-art method (DeViSE) on the same zero-shot benchmark.
• The method achieves competitive performance on the original 1,000-class ImageNet task, with ConSE(1) matching the Softmax baseline at 55.6% hit@1, while still enabling zero-shot generalization.
• The model’s embedding magnitude implicitly reflects prediction confidence: embeddings with lower L2 norms correspond to uncertain or semantically diverse predictions.
• The ConSE(1) model’s hit@1 score differs slightly from the Softmax baseline (55.1% vs. 55.6%) due to a non-uniform treatment of synonym word vectors during ranking, despite identical input scores.
• Performance degrades with increasing distance from training classes in the ImageNet hierarchy, but remains robust and superior to prior methods under such conditions.
• The method is robust to the choice of visual and text models, as it only requires relative class scores and a semantic space where similar concepts are close.