[Paper Review] Global Semantic Consistency for Zero-Shot Learning

Fan Wu, Kai Tian+2 more|arXiv (Cornell University)|Jun 22, 2018

Domain Adaptation and Few-Shot Learning29 references3 citations

TL;DR

This paper proposes Global Semantic Consistency Network (GSC-Net), an end-to-end deep learning framework for zero-shot learning that leverages both seen and unseen class semantic embeddings via a global consistency regularization in the final fully connected layer. By integrating soft label embedding loss and a parametric novelty detection mechanism, GSC-Net achieves state-of-the-art performance on ZSL and GZSL benchmarks across three visual attribute datasets, significantly improving generalization to unseen classes.

ABSTRACT

In image recognition, there are many cases where training samples cannot cover all target classes. Zero-shot learning (ZSL) utilizes the class semantic information to classify samples of the unseen categories that have no corresponding samples contained in the training set. In this paper, we propose an end-to-end framework, called Global Semantic Consistency Network (GSC-Net for short), which makes complete use of the semantic information of both seen and unseen classes, to support effective zero-shot learning. We also adopt a soft label embedding loss to further exploit the semantic relationships among classes. To adapt GSC-Net to a more practical setting, Generalized Zero-shot Learning (GZSL), we introduce a parametric novelty detection mechanism. Our approach achieves the state-of-the-art performance on both ZSL and GZSL tasks over three visual attribute datasets, which validates the effectiveness and advantage of the proposed framework.

Motivation & Objective

• To address the limitation of existing ZSL methods that fail to effectively utilize semantic information from both seen and unseen classes during training.
• To improve zero-shot generalization by enforcing global semantic consistency in the final fully connected layer of a CNN.
• To enhance feature learning through a soft label embedding loss that models semantic relationships among classes.
• To enable effective adaptation to the generalized zero-shot learning (GZSL) setting via a learnable novelty detection mechanism.
• To validate the effectiveness of using the final FC layer weights as semantic embeddings for all classes.

Proposed method

• GSC-Net integrates a global semantic consistency regularization by fixing the final fully connected layer’s weight to the class attribute matrix, enabling end-to-end training with full semantic supervision.
• The framework employs a neural weighted unit to enhance feature learning under global semantic constraints.
• A soft label embedding loss is introduced to model inter-class semantic relationships, with a hyperparameter α controlling the supervision strength on unseen classes.
• A parametric novelty detection mechanism is proposed for GZSL, using a learnable threshold γ to distinguish between seen and unseen class predictions.
• The model is trained using cross-entropy loss over all classes (seen and unseen) during training, enabling consistent inference in both ZSL and GZSL settings.
• The final FC layer weights are empirically validated as effective semantic embeddings via t-SNE visualization on CIFAR-100.

Experimental results

Research questions

• RQ1 Can enforcing global semantic consistency in the final fully connected layer improve zero-shot generalization?
• RQ2 How does incorporating soft label embedding loss enhance knowledge transfer to unseen classes?
• RQ3 Can a learnable novelty detection mechanism effectively balance performance on seen and unseen classes in GZSL?
• RQ4 Is the final fully connected layer weight matrix a valid and effective representation of class semantic embeddings?
• RQ5 Does the proposed end-to-end framework outperform existing SOTA methods on both ZSL and GZSL benchmarks?

Key findings

• GSC-Net-SLE-PND achieves state-of-the-art performance on all three datasets (CUB, SUN, AWA2) for both ZSL and GZSL tasks.
• On the CUB dataset, the harmonic mean (H) score improved from 34.4% to 55.4% in GZSL, demonstrating strong balance between seen and unseen class performance.
• On AWA2, which has a highly imbalanced class distribution, GSC-Net-SLE-PND significantly outperforms all baselines, especially in unseen class accuracy.
• The t-SNE visualization confirms that the final FC layer weights form a meaningful semantic space where semantically similar classes are spatially close.
• The model achieves high accuracy within 20 epochs and generalization performance correlates strongly with validation accuracy, supporting early stopping.
• The framework is consistent with standard supervised learning, enabling seamless integration of new unseen samples into training without architectural changes.