[Paper Review] Continual Learning for Text Classification with Information Disentanglement Based Regularization

Yufan Huang, Yanzhe Zhang+3 more|arXiv (Cornell University)|Apr 12, 2021

Domain Adaptation and Few-Shot Learning43 references4 citations

TL;DR

This paper proposes an information disentanglement-based regularization (IDBR) method for continual text classification that separates hidden representations into task-generic and task-specific components, applying distinct regularization to preserve long-term knowledge while allowing flexibility for new tasks. Experiments show IDBR outperforms state-of-the-art baselines on five benchmark datasets with reduced catastrophic forgetting.

ABSTRACT

Continual learning has become increasingly important as it enables NLP models to constantly learn and gain knowledge over time. Previous continual learning methods are mainly designed to preserve knowledge from previous tasks, without much emphasis on how to well generalize models to new tasks. In this work, we propose an information disentanglement based regularization method for continual learning on text classification. Our proposed method first disentangles text hidden spaces into representations that are generic to all tasks and representations specific to each individual task, and further regularizes these representations differently to better constrain the knowledge required to generalize. We also introduce two simple auxiliary tasks: next sentence prediction and task-id prediction, for learning better generic and specific representation spaces. Experiments conducted on large-scale benchmarks demonstrate the effectiveness of our method in continual text classification tasks with various sequences and lengths over state-of-the-art baselines. We have publicly released our code at https://github.com/GT-SALT/IDBR.

Motivation & Objective

• To address catastrophic forgetting in continual text classification by distinguishing task-generic and task-specific knowledge.
• To improve model generalization to new tasks while preserving knowledge from previous tasks.
• To reduce memory and training overhead compared to replay-based methods.
• To develop a regularization strategy that treats generic and specific representations differently for optimal knowledge retention.
• To demonstrate effectiveness across diverse task orders and sequence lengths on large-scale benchmarks.

Proposed method

• Disentangles BERT-based hidden representations into task-generic and task-specific components using auxiliary tasks.
• Uses next sentence prediction to learn task-generic representations and task-id prediction to learn task-specific representations.
• Applies different regularization constraints: stronger on task-generic space, lighter on task-specific space to preserve stability.
• Introduces a memory selection rule (K-Means) to store only 1% of the most representative examples from previous tasks.
• Combines regularization with a small episodic memory buffer to enhance knowledge recovery.
• Uses a dual-branch head for classification, with shared and task-specific heads for generic and specific representations.

Experimental results

Research questions

• RQ1 Can disentangling hidden representations into generic and specific components improve continual learning in text classification?
• RQ2 Does applying different regularization strengths to generic and specific representations reduce catastrophic forgetting?
• RQ3 Can auxiliary tasks like next sentence prediction and task-id prediction effectively guide disentanglement?
• RQ4 How does memory selection via K-Means impact performance under limited memory budgets?
• RQ5 Does the method generalize across different task orders and sequence lengths?

Key findings

• IDBR achieves an average accuracy of 73.72% across five benchmarks, outperforming state-of-the-art methods.
• The ablation study shows that combining both auxiliary tasks (next sentence and task-id prediction) yields the highest performance gain.
• Regularization on both generic and specific spaces is essential, with stronger regularization on the generic space yielding greater forgetting reduction.
• K-Means memory selection improves performance over random sampling, especially on challenging task orders.
• Forgetting measures drop significantly with IDBR, with a 2.89 average forgetting score after five tasks, indicating reduced catastrophic forgetting.
• The method maintains strong performance across various task sequences and input lengths, demonstrating robustness and generalization.