[Paper Review] Adapting BERT for Continual Learning of a Sequence of Aspect Sentiment Classification Tasks

Zixuan Ke, Xu Hu+1 more|arXiv (Cornell University)|Dec 5, 2021

Domain Adaptation and Few-Shot Learning10 citations

TL;DR

This paper proposes B-CL, a novel BERT-based continual learning framework for aspect sentiment classification (ASC) that uses capsule-based continual learning adapters (CLA) to enable forward and backward knowledge transfer while preventing catastrophic forgetting. By leveraging dynamic routing for shared knowledge extraction and task-specific masks for isolation, B-CL achieves state-of-the-art performance, outperforming all baselines by significant margins on both new and old tasks in a sequence of 19 ASC tasks.

ABSTRACT

This paper studies continual learning (CL) of a sequence of aspect sentiment classification (ASC) tasks. Although some CL techniques have been proposed for document sentiment classification, we are not aware of any CL work on ASC. A CL system that incrementally learns a sequence of ASC tasks should address the following two issues: (1) transfer knowledge learned from previous tasks to the new task to help it learn a better model, and (2) maintain the performance of the models for previous tasks so that they are not forgotten. This paper proposes a novel capsule network based model called B-CL to address these issues. B-CL markedly improves the ASC performance on both the new task and the old tasks via forward and backward knowledge transfer. The effectiveness of B-CL is demonstrated through extensive experiments.

Motivation & Objective

• To address the challenge of continual learning in aspect sentiment classification (ASC), where models must learn a sequence of domain-specific sentiment tasks without forgetting previous knowledge.
• To enable forward knowledge transfer from previous tasks to accelerate learning on new tasks, even without access to old training data.
• To prevent catastrophic forgetting by isolating task-specific knowledge using task masks while preserving performance on previously learned tasks.
• To propose a novel continual learning adapter (CLA) that integrates capsule networks and dynamic routing into BERT for effective knowledge transfer and retention.
• To demonstrate that fine-tuned BERT alone fails in continual learning due to highly task-specific representations, necessitating a new architecture like B-CL.

Proposed method

• B-CL employs a continual learning adapter (CLA) built on capsule networks with dynamic routing to identify and transfer shared knowledge from previous tasks to the current task.
• The CLA uses routing mechanisms to group attention heads based on similarity, enabling selective transfer of relevant knowledge from old tasks to the new task.
• Task-specific modules (TSM) are introduced to isolate task-specific representations, preventing interference during learning of new tasks.
• Knowledge sharing modules (KSM) are used to aggregate and transfer shared knowledge across tasks, improving generalization and learning efficiency.
• The model is trained end-to-end with a multi-task loss that balances performance on current and previous tasks, using task masks to protect specific knowledge.
• B-CL is integrated into a pre-trained BERT backbone, allowing it to leverage transfer learning while maintaining continual learning capabilities.

Experimental results

Research questions

• RQ1 Can a BERT-based model effectively transfer shared knowledge from previous aspect sentiment classification tasks to accelerate learning on new tasks?
• RQ2 Can a continual learning framework prevent catastrophic forgetting in aspect sentiment classification when tasks are learned sequentially?
• RQ3 Does the use of capsule networks and dynamic routing in a continual learning adapter improve knowledge transfer and retention compared to standard fine-tuning or regularization methods?
• RQ4 How does B-CL compare to existing continual learning baselines (e.g., EWC, OWM, HAT) in terms of performance on both new and previously learned ASC tasks?
• RQ5 To what extent does the proposed architecture support both forward and backward knowledge transfer in a sequence of 19 diverse ASC tasks?

Key findings

• B-CL achieves an average Macro-F1 score of 0.8140 across 19 ASC tasks, significantly outperforming the next best baseline (OWM) with 0.7931.
• The forward transfer performance of B-CL (0.7993 Macro-F1) exceeds all other baselines' forward transfer results, demonstrating strong knowledge transfer capability.
• The ablation study confirms that both the knowledge sharing module (KSM) and task-specific module (TSM) are essential, as removing either reduces performance by over 8% in Macro-F1.
• Fine-tuned BERT and Adapter-BERT perform poorly under continual learning (WDF), with Macro-F1 dropping to 0.4308 and 0.4481 respectively, due to catastrophic forgetting from task-specific representations.
• B-CL’s backward transfer improves performance slightly from 0.7993 to 0.8140 Macro-F1, indicating that continued training further refines model generalization.
• The results show that BERT fine-tuning fails in continual learning due to highly task-specific knowledge, validating the need for a new architecture like B-CL.