[Paper Review] Zero-Shot Learning Through Cross-Modal Transfer

Richard Socher, Milind Ganjoo+4 more|arXiv (Cornell University)|Jan 16, 2013

Domain Adaptation and Few-Shot Learning33 references805 citations

TL;DR

This paper proposes a zero-shot learning model that leverages unsupervised word vectors and visual feature embeddings to classify both seen and unseen object categories without any training data for unseen classes. By mapping images into a semantic space learned from text and using outlier detection to distinguish known from unseen classes, the model achieves state-of-the-art performance on seen classes (up to 80% accuracy) and reasonable performance on unseen classes (30–15% accuracy), even without manually defined attributes.

ABSTRACT

This work introduces a model that can recognize objects in images even if no training data is available for the objects. The only necessary knowledge about the unseen categories comes from unsupervised large text corpora. In our zero-shot framework distributional information in language can be seen as spanning a semantic basis for understanding what objects look like. Most previous zero-shot learning models can only differentiate between unseen classes. In contrast, our model can both obtain state of the art performance on classes that have thousands of training images and obtain reasonable performance on unseen classes. This is achieved by first using outlier detection in the semantic space and then two separate recognition models. Furthermore, our model does not require any manually defined semantic features for either words or images.

Motivation & Objective

• To enable zero-shot recognition of visual objects without any training images for unseen classes.
• To bridge visual and semantic modalities using unsupervised word vectors from large text corpora.
• To unify seen and unseen class classification within a single probabilistic framework.
• To eliminate the need for manually defined visual or semantic attributes in zero-shot learning.

Proposed method

• Images are embedded into a low-dimensional semantic space using deep neural networks trained on image features.
• Word vectors are learned from large, unsupervised text corpora to represent semantic similarity.
• A mixture of isotropic Gaussians models the distribution of seen-class features in the semantic space.
• Outlier detection via marginal probability thresholding determines whether a test image belongs to a seen or unseen class.
• For seen classes, a softmax classifier is applied on original image features.
• For unseen classes, classification is performed using Gaussian likelihoods centered on semantic word vectors.

Experimental results

Research questions

• RQ1 Can a model achieve high accuracy on seen classes while also performing reasonably on unseen classes in zero-shot learning?
• RQ2 How effective is unsupervised word vector representation in enabling cross-modal transfer for zero-shot visual recognition?
• RQ3 Can outlier detection in the semantic space reliably separate known from unknown visual categories?
• RQ4 To what extent does the absence of manual attribute engineering affect performance in zero-shot learning?
• RQ5 How does the model's performance vary with different thresholds for outlier detection?

Key findings

• The model achieves up to 80% accuracy on seen classes when using an optimal outlier detection threshold.
• On unseen classes, the model attains classification accuracy between 30% and 15%, significantly above random chance (10%).
• Performance on zero-shot classes is highest when the unseen classes have semantic and visual similarities to seen classes (e.g., cat and truck), enabling effective transfer.
• When zero-shot classes are semantically and visually dissimilar to seen classes (e.g., cat and dog), performance drops close to random.
• The model achieves up to 90% accuracy in differentiating between zero-shot classes alone, demonstrating strong zero-shot discrimination capability.