[Paper Review] Joint Dictionaries for Zero-Shot Learning

Soheil Kolouri, Mohammad Rostami+2 more|arXiv (Cornell University)|Sep 12, 2017

Domain Adaptation and Few-Shot Learning36 references3 citations

TL;DR

This paper proposes Joint Dictionaries for Zero-Shot Learning (JDZSL), a novel method that jointly learns visual and attribute dictionaries with shared sparse coding to mitigate domain shift and hubness in zero-shot classification. By incorporating entropy regularization and a transductive label propagation scheme, JDZSL achieves state-of-the-art or competitive performance on AWA, SUN, and CUB200-2011 datasets, significantly improving zero-shot accuracy over baseline methods.

ABSTRACT

A classic approach toward zero-shot learning (ZSL) is to map the input domain to a set of semantically meaningful attributes that could be used later on to classify unseen classes of data (e.g. visual data). In this paper, we propose to learn a visual feature dictionary that has semantically meaningful atoms. Such dictionary is learned via joint dictionary learning for the visual domain and the attribute domain, while enforcing the same sparse coding for both dictionaries. Our novel attribute aware formulation provides an algorithmic solution to the domain shift/hubness problem in ZSL. Upon learning the joint dictionaries, images from unseen classes can be mapped into the attribute space by finding the attribute aware joint sparse representation using solely the visual data. We demonstrate that our approach provides superior or comparable performance to that of the state of the art on benchmark datasets.

Motivation & Objective

• To address the domain shift and hubness problems in zero-shot learning (ZSL) by modeling the relationship between visual features and semantic attributes more robustly.
• To improve zero-shot classification accuracy by learning semantically meaningful visual and attribute dictionaries through joint sparse coding.
• To reduce the distributional discrepancy between seen and unseen classes by enforcing consistency in sparse representations across domains.
• To enhance performance using a transductive learning scheme that propagates labels on predicted attributes, reducing hubness effects.
• To provide a unified, attribute-aware formulation of joint dictionary learning that outperforms existing ZSL methods on standard benchmarks.

Proposed method

• The method formulates zero-shot learning as a joint dictionary learning problem where visual and attribute dictionaries are trained simultaneously with shared sparse coding constraints.
• An entropy regularization term is introduced to the optimization to encourage balanced and meaningful sparse codes, reducing domain shift and improving clustering of attributes.
• The approach uses a transductive label propagation step on the graph of predicted attributes to refine predictions and alleviate hubness in the embedding space.
• The model leverages VGG19 features for images and human-annotated or word2vec-based attributes, with parameters tuned via k-fold cross-validation.
• The joint dictionary learning framework is optimized using alternating minimization with regularization terms for sparsity, consistency, and entropy.
• The final prediction for unseen classes is made by computing the attribute-aware joint sparse representation of a test image using only visual features.

Experimental results

Research questions

• RQ1 Can joint dictionary learning with shared sparse coding improve zero-shot classification by reducing domain shift between seen and unseen classes?
• RQ2 How does entropy regularization in the sparse coding process affect the clustering quality and generalization of semantic attributes?
• RQ3 To what extent does a transductive label propagation step on predicted attributes mitigate the hubness problem in zero-shot learning?
• RQ4 Does the proposed attribute-aware joint dictionary learning framework achieve state-of-the-art performance across diverse zero-shot benchmarks?
• RQ5 How does the method compare to existing ZSL approaches when using the same visual features and attributes?

Key findings

• The AAw formulation with entropy regularization significantly improves attribute clustering and reduces domain shift, as evidenced by clearer t-SNE visualizations in Figure 2.
• The transductive label propagation step (TAA formulation) further boosts classification accuracy by mitigating hubness, particularly when combined with the AAw formulation.
• JDZSL achieves state-of-the-art or competitive hit@1 accuracy on the AWA, SUN, and CUB200-2011 datasets, outperforming or matching recent SOTA methods.
• The method demonstrates consistent performance across diverse zero-shot recognition tasks, including object and scene recognition, indicating strong generalization.
• The ablation study confirms that both entropy regularization and transductive label propagation are critical components, with the combination yielding the highest performance gains.
• The approach maintains strong performance across all three benchmark datasets, suggesting robustness to dataset-specific variations in attributes and features.