[Paper Review] Recovering the Missing Link: Predicting Class-Attribute Associations for Unsupervised Zero-Shot Learning

Ziad Al-Halah, Makarand Tapaswi+1 more|arXiv (Cornell University)|Oct 15, 2016

Domain Adaptation and Few-Shot Learning29 references22 citations

TL;DR

This paper proposes CAAP, a novel method for unsupervised zero-shot learning that automatically predicts class-attribute associations using only the name of an unseen class. By learning semantic relationships in a shared embedding space, CAAP predicts attributes without manual supervision and achieves state-of-the-art performance on both AwA and aPascal/aYahoo datasets, outperforming prior methods by over 18% on aPascal/aYahoo.

ABSTRACT

Collecting training images for all visual categories is not only expensive but also impractical. Zero-shot learning (ZSL), especially using attributes, offers a pragmatic solution to this problem. However, at test time most attribute-based methods require a full description of attribute associations for each unseen class. Providing these associations is time consuming and often requires domain specific knowledge. In this work, we aim to carry out attribute-based zero-shot classification in an unsupervised manner. We propose an approach to learn relations that couples class embeddings with their corresponding attributes. Given only the name of an unseen class, the learned relationship model is used to automatically predict the class-attribute associations. Furthermore, our model facilitates transferring attributes across data sets without additional effort. Integrating knowledge from multiple sources results in a significant additional improvement in performance. We evaluate on two public data sets: Animals with Attributes and aPascal/aYahoo. Our approach outperforms state-of-the-art methods in both predicting class-attribute associations and unsupervised ZSL by a large margin.

Motivation & Objective

• To eliminate the need for manual annotation of class-attribute associations in zero-shot learning.
• To enable automatic prediction of attributes for unseen classes using only their names.
• To facilitate transfer of visual attributes across datasets without additional labeling.
• To improve zero-shot classification performance through joint modeling of multiple semantic relations.
• To minimize user intervention by relying solely on class names and pre-trained word embeddings.

Proposed method

• The method models class-attribute associations as semantic relations in a shared embedding space using a multi-relational link prediction framework.
• It leverages pre-trained word embeddings (e.g., from GloVe or Word2Vec) to represent class and attribute names as vectors.
• A differentiable scoring function computes the likelihood of a relation (e.g., 'has_color') between a class and an attribute via dot product in the embedding space.
• The model is trained end-to-end using negative sampling and contrastive loss to optimize relation predictions.
• It supports joint learning of multiple relations, including attribute-based and hierarchical relations (e.g., 'has_ancestor').
• The framework enables zero-shot transfer of attributes across datasets by reusing learned relations and embeddings.

Experimental results

Research questions

• RQ1 Can we predict class-attribute associations for unseen classes without requiring manual attribute annotations?
• RQ2 Can a model trained on one dataset generalize to predict attributes for classes in another dataset with no additional effort?
• RQ3 Does joint modeling of multiple semantic relations (e.g., attributes and hierarchies) improve zero-shot classification performance?
• RQ4 How does the performance of the proposed method compare to state-of-the-art unsupervised ZSL approaches?
• RQ5 Can the model automatically infer hierarchical relationships (e.g., ancestors) from class names alone?

Key findings

• CAAP achieves 68.6% zero-shot learning accuracy on the AwA dataset, outperforming the previous state-of-the-art by 8.5%.
• On the aPascal/aYahoo dataset, CAAP reaches 49.0% accuracy, a significant 18.8% improvement over the next best method.
• The model predicts attribute associations for unseen classes with high accuracy, even for non-animal categories like 'jetski' and 'carriage'.
• Cross-dataset attribute transfer boosts performance by enlarging the description vocabulary, leading to more discriminative classifiers.
• The model achieves 89.8% mAP in predicting hierarchical ancestor relations, demonstrating strong generalization to semantic hierarchies.
• Joint learning of attribute and hierarchical relations improves attribute prediction performance, with gains of up to 2.5% on 'has_pattern' and 2.1% on 'feeding_type'.