[Paper Review] Preserving Semantic Relations for Zero-Shot Learning

Yashas Annadani, Soma Biswas|arXiv (Cornell University)|Mar 8, 2018

Domain Adaptation and Few-Shot Learning34 references79 citations

TL;DR

The paper proposes preserving semantic relations among classes in an embedding space by modeling identical, similar, and dissimilar relationships via a relation-aware encoder–decoder framework, achieving state-of-the-art results in several zero-shot and generalized zero-shot benchmarks and enabling approximate semantic inferences when some class embeddings are unavailable.

ABSTRACT

Zero-shot learning has gained popularity due to its potential to scale recognition models without requiring additional training data. This is usually achieved by associating categories with their semantic information like attributes. However, we believe that the potential offered by this paradigm is not yet fully exploited. In this work, we propose to utilize the structure of the space spanned by the attributes using a set of relations. We devise objective functions to preserve these relations in the embedding space, thereby inducing semanticity to the embedding space. Through extensive experimental evaluation on five benchmark datasets, we demonstrate that inducing semanticity to the embedding space is beneficial for zero-shot learning. The proposed approach outperforms the state-of-the-art on the standard zero-shot setting as well as the more realistic generalized zero-shot setting. We also demonstrate how the proposed approach can be useful for making approximate semantic inferences about an image belonging to a category for which attribute information is not available.

Motivation & Objective

• Motivate zero-shot learning to recognize novel categories without new labeled data by leveraging semantic descriptions (attributes/word embeddings).
• Preserve the structure of the semantic space in the image embedding space by decomposing it into identical, similar, and dissimilar class relations.
• Develop a trainable encoder–decoder model that enforces relation-aware objectives and reconstruction to improve ZSL and generalized ZSL performance.
• Show that preserving semantic relations improves discrimination for seen classes while generalizing to unseen classes, and enable approximate semantic inferences when some embeddings are missing.

Proposed method

• Use an encoder f(y; theta_f) to map class embeddings to the visual (embedding) space (visual space) and a decoder g(x; theta_g) to reconstruct inputs.
• Decompose semantic space into relations: identical (delta=1), semantically similar (tau <= delta < 1), semantically dissimilar (delta < tau) based on cosine similarity delta between class embeddings.
• Introduce three objective terms: O1 aligns identical and dissimilar pairs via an adaptive hinge-like loss; O2 enforces similarity for semantically similar classes while respecting the limit with respect to dissimilar classes; O3 reconstruction loss ||y_r - hat{y}_r||^2 to regularize the encoder.
• Use a quadruplet-like sampling (y_r, x_i, x_j, x_k) to encode identical, similar, and dissimilar relations; online hard negative mining selects informative tuples for updates.
• Train with a combined objective O = (1/|B|) sum_B (O1 + lambda1 * O2 + lambda2 * O3) and perform nearest-neighbor inference in the visual space using s(f(y^c_r), x^u) for class c_r.
• Mine tuples efficiently by sampling p candidates for x_j (similar) and x_k (dissimilar) and updating with the highest loss terms to improve convergence without hard-mining across all candidates.

Experimental results

Research questions

• RQ1 Does preserving the semantic structure of class embeddings in the embedding space improve zero-shot recognition performance?
• RQ2 Can a relation-aware encoder–decoder framework outperform state-of-the-art ZSL methods on standard and generalized ZSL benchmarks?
• RQ3 How does the proposed approach perform on large-scale datasets (e.g., ImageNet) and support approximate semantic inferences when some class embeddings are unavailable?
• RQ4 What is the contribution of each component (relation preservation, reconstruction loss) to overall performance?

Key findings

• Achieves state-of-the-art results on SUN, AWA2, and CUB in conventional ZSL settings.
• Obtains strong generalized ZSL performance with competitive harmonic means across SUN, AWA2, CUB, and aPY datasets.
• Outperforms competing methods on large-scale ImageNet in both conventional and generalized ZSL settings.
• Demonstrates approximate semantic inference: even when a category’s embeddings are unavailable, the model can suggest semantically related known categories based on cosine similarity.
• Tuple mining with the proposed O1–O3 framework provides faster convergence and robust updates (early gains within ~5 epochs).
• Across baselines, preserving semantic relations (O2) yields noticeable gains, especially in coarse-grained datasets where inter-class semantics are wider.