[Paper Review] Online Lifelong Generalized Zero-Shot Learning

Chandan Gautam, Sethupathy Parameswaran+2 more|arXiv (Cornell University)|Mar 19, 2021

Domain Adaptation and Few-Shot Learning43 references4 citations

TL;DR

This paper proposes the first task-free continual generalized zero-shot learning (CZSL) framework that enables models to learn from streaming data without task boundary information. By combining variational autoencoders, experience replay, knowledge distillation using dark knowledge, and short-term memory, the method achieves state-of-the-art performance on five benchmark datasets with minimal catastrophic forgetting, outperforming prior methods by up to 9% in harmonic accuracy.

ABSTRACT

Methods proposed in the literature for zero-shot learning (ZSL) are typically suitable for offline learning and cannot continually learn from sequential streaming data. The sequential data comes in the form of tasks during training. Recently, a few attempts have been made to handle this issue and develop continual ZSL (CZSL) methods. However, these CZSL methods require clear task-boundary information between the tasks during training, which is not practically possible. This paper proposes a task-free (i.e., task-agnostic) CZSL method, which does not require any task information during continual learning. The proposed task-free CZSL method employs a variational autoencoder (VAE) for performing ZSL. To develop the CZSL method, we combine the concept of experience replay with knowledge distillation and regularization. Here, knowledge distillation is performed using the training sample's dark knowledge, which essentially helps overcome the catastrophic forgetting issue. Further, it is enabled for task-free learning using short-term memory. Finally, a classifier is trained on the synthetic features generated at the latent space of the VAE. Moreover, the experiments are conducted in a challenging and practical ZSL setup, i.e., generalized ZSL (GZSL). These experiments are conducted for two kinds of single-head continual learning settings: (i) mild setting-: task-boundary is known only during training but not during testing; (ii) strict setting-: task-boundary is not known at training, as well as testing. Experimental results on five benchmark datasets exhibit the validity of the approach for CZSL.

Motivation & Objective

• To address the limitations of conventional zero-shot learning (ZSL) and continual learning, which cannot handle sequential data or prevent catastrophic forgetting.
• To develop a continual ZSL framework that operates without task boundary information during training or inference—i.e., a true task-free setting.
• To enable lifelong learning for ZSL in practical, real-world scenarios where data arrives sequentially and task identities are unknown.
• To propose a novel evaluation protocol suitable for task-free CZSL, as existing benchmarks assume task boundaries.
• To minimize catastrophic forgetting while maintaining high performance on both seen and unseen classes in generalized ZSL settings.

Proposed method

• Uses a variational autoencoder (VAE) to generate synthetic visual features from semantic class descriptors, enabling zero-shot classification in the latent space.
• Employs experience replay with a memory buffer (M) to store past data samples, allowing joint training with current data to mitigate forgetting.
• Applies knowledge distillation using 'dark knowledge' from training samples to transfer model capacity and improve generalization without a student-teacher network.
• Introduces a short-term memory (M_st) to store recent samples, enabling effective task-free learning by maintaining temporal context.
• Trains a classifier on the latent features generated by the VAE, with regularization from replay and distillation to stabilize learning.
• Supports two training strategies: one using a fixed memory (M_b) and another using a dynamic short-term memory (M_st), with the latter showing superior performance.

Experimental results

Research questions

• RQ1 Can a continual zero-shot learning framework be developed that does not require task boundary information during training or inference?
• RQ2 How can catastrophic forgetting be effectively mitigated in a continual generalized zero-shot learning setup without task annotations?
• RQ3 What is the impact of experience replay and knowledge distillation using dark knowledge on performance in a task-free CZSL setting?
• RQ4 How do memory size and latent dimensionality affect model performance in the proposed task-free CZSL framework?
• RQ5 To what extent does the proposed method approach the upper bound performance in generalized zero-shot learning under strict continual learning conditions?

Key findings

• The proposed Tf-GCZSL method with short-term memory (M_st) outperforms the baseline method (M_b) by more than 5% in harmonic accuracy on CUB, aPY, AWA1, AWA2, and SUN datasets.
• Tf-GCZSL achieves a harmonic accuracy within 8.55% of the upper bound on CUB, 9.28% on aPY, 4.38% on AWA1, 1.49% on AWA2, and 8.14% on SUN, demonstrating strong performance close to the theoretical limit.
• Knowledge distillation using dark knowledge significantly improves performance, especially when combined with short-term memory, leading to better generalization and reduced forgetting.
• Performance improves with increasing memory size and optimal latent dimensionality—too small or too large dimensions degrade feature quality and model performance.
• The ablation study confirms that the proposed method consistently outperforms sequential baselines across all hyperparameter settings on the CUB dataset.
• The framework is robust to catastrophic forgetting, as joint training with replayed samples regularizes the model and prevents overfitting to recent data.