[Paper Review] Bookworm continual learning: beyond zero-shot learning and continual learning

Kai Wang, Luis Herranz+2 more|arXiv (Cornell University)|Jun 26, 2020

Domain Adaptation and Few-Shot Learning15 references4 citations

TL;DR

This paper introduces Bookworm Continual Learning (BCL), a unified framework that generalizes both continual learning and zero-shot learning by enabling continual visual model updates with a fixed semantic model. It proposes the bidirectional imagination (BImag) framework with conditional VAEs to generate features for past and future classes, showing that joint class-attribute conditioning and asymmetric generation significantly reduce forgetting and improve performance over baselines on CUB and AwA datasets.

ABSTRACT

We propose bookworm continual learning(BCL), a flexible setting where unseen classes can be inferred via a semantic model, and the visual model can be updated continually. Thus BCL generalizes both continual learning (CL) and zero-shot learning (ZSL). We also propose the bidirectional imagination (BImag) framework to address BCL where features of both past and future classes are generated. We observe that conditioning the feature generator on attributes can actually harm the continual learning ability, and propose two variants (joint class-attribute conditioning and asymmetric generation) to alleviate this problem.

Motivation & Objective

• To address the limitations of static visual recognition models that assume all classes are known in advance and cannot adapt to new concepts.
• To unify continual learning (CL) and zero-shot learning (ZSL) into a more realistic, dynamic setting where new classes are learned over time using semantic priors.
• To investigate how semantic models affect continual learning, particularly whether attribute-based conditioning helps or harms continual learning performance.
• To develop a memory- and computation-efficient framework that enables joint learning of past and future class features for continual visual recognition.

Proposed method

• Proposes Bookworm Continual Learning (BCL), a generalized continual learning setting where a fixed semantic model enables recognition of unseen classes, and the visual model is updated continually over time.
• Introduces the bidirectional imagination (BImag) framework, which uses a conditional VAE to generate visual features for both past and future classes using class and attribute information.
• Employs feature distillation and replay alignment to preserve knowledge from previous tasks and mitigate catastrophic forgetting.
• Proposes two variants—joint class-attribute conditioning and asymmetric generation—to address the negative impact of attribute conditioning on continual learning performance.
• Uses a novel evaluation metric, average AUTAC (area under task-accuracy curve), to assess performance across all time steps in the continual learning sequence.
• Trains and evaluates the model using a unified framework that jointly optimizes visual feature extraction, feature generation, and classification.

Experimental results

Research questions

• RQ1 Can a unified framework effectively combine continual learning and zero-shot learning to enable recognition of both seen and unseen classes over time?
• RQ2 Does conditioning the feature generator on attributes improve or impair continual learning performance, and why?
• RQ3 How can backward (past) and forward (future) feature generation be balanced to reduce forgetting and improve generalization?
• RQ4 Can joint conditioning on class and attributes, or asymmetric generation, mitigate the negative impact of attribute-based semantic models on continual learning?

Key findings

• The attr-BImag variant, which conditions on attributes, performs worse than the class-BImag baseline in continual learning, indicating that attribute conditioning can impair forgetting mitigation.
• The class-attr-BImag variant, which conditions on both class and attributes, significantly improves performance over attr-BImag, achieving 0.600 mean AUTAC on CUB 150/50 and 0.750 on AwA 40/10.
• The asymmetric generation variant (asym-BImag) achieves the best performance on CUB 150/50 with a mean AUTAC of 0.588, outperforming both attr-BImag and class-attr-BImag in this setting.
• On AwA 40/10, class-attr-BImag achieves the highest mean AUTAC of 0.750, demonstrating superior generalization across tasks.
• The BImag framework achieves competitive results in the GZSL setting (t=1), with 0.578 mean AUTAC on AwA 40/10, second only to f-VAEGAN-D2.
• The proposed BCL framework outperforms standard CL (class-BImag) at initial time steps due to the integration of semantic priors, while also improving over GZSL through continual model updates.