[Paper Review] Preserving Earlier Knowledge in Continual Learning with the Help of All Previous Feature Extractors

Zhuoyun Li, Changhong Zhong+3 more|arXiv (Cornell University)|Apr 28, 2021

Domain Adaptation and Few-Shot Learning40 references21 citations

TL;DR

This paper proposes a continual learning framework that preserves earlier knowledge by fusing all previously learned feature extractors into a unified classifier, using learnable feature transformations and dynamic pruning to control model size. The method achieves state-of-the-art performance on class-incremental learning benchmarks by significantly reducing catastrophic forgetting, especially for early-learned classes.

ABSTRACT

Continual learning of new knowledge over time is one desirable capability for intelligent systems to recognize more and more classes of objects. Without or with very limited amount of old data stored, an intelligent system often catastrophically forgets previously learned old knowledge when learning new knowledge. Recently, various approaches have been proposed to alleviate the catastrophic forgetting issue. However, old knowledge learned earlier is commonly less preserved than that learned more recently. In order to reduce the forgetting of particularly earlier learned old knowledge and improve the overall continual learning performance, we propose a simple yet effective fusion mechanism by including all the previously learned feature extractors into the intelligent model. In addition, a new feature extractor is included to the model when learning a new set of classes each time, and a feature extractor pruning is also applied to prevent the whole model size from growing rapidly. Experiments on multiple classification tasks show that the proposed approach can effectively reduce the forgetting of old knowledge, achieving state-of-the-art continual learning performance.

Motivation & Objective

• To address catastrophic forgetting in continual learning, particularly the disproportionate forgetting of knowledge learned in earlier rounds.
• To improve overall continual learning performance by leveraging knowledge from all previous models, not just the most recent one.
• To maintain model efficiency despite incremental feature extractor addition through dynamic pruning.
• To investigate whether fusing multiple past feature extractors is more effective than knowledge distillation from a single model.

Proposed method

• A knowledge fusion mechanism combines feature representations from all previously trained feature extractors, each followed by a learnable transformation layer to align different feature spaces.
• The fused features are used as input to a shared classifier head during training on new classes, enabling the model to access knowledge from all prior learning rounds.
• A feature extractor pruning strategy is applied after each new model is trained, reducing kernel and parameter counts by 50–60% and 70% respectively to limit model growth.
• All previous feature extractors are kept frozen during training to preserve learned knowledge and prevent interference.
• The framework is evaluated using standard class-incremental learning protocols on CIFAR-100 and other datasets with ResNet and other CNN backbones.
• Ablation studies compare the impact of fusion, feature transformation, and pruning, validating each component’s contribution.

Experimental results

Research questions

• RQ1 Does fusing knowledge from all previous feature extractors reduce forgetting of early-learned classes more effectively than relying on only the most recent model?
• RQ2 Can the inclusion of multiple past feature extractors improve continual learning performance compared to single-model distillation?
• RQ3 How effective is the proposed pruning mechanism in controlling model size without sacrificing performance?
• RQ4 Is the performance gain due to increased model capacity or due to preserved knowledge from fixed feature extractors?

Key findings

• The proposed method achieved 70.5% mean accuracy on CIFAR-100 after 10 learning rounds, outperforming strong baselines and setting a new state-of-the-art.
• The inclusion of all previous feature extractors reduced forgetting of early-learned classes, as shown by a 10–15% performance gain on classes learned in round 1 compared to methods relying only on recent models.
• Ablation studies confirmed that all components—fusion, feature transformation, and pruning—were necessary, with the full method achieving 60.5% accuracy on individual old classes versus 50.7% without any components.
• Removing a single past feature extractor led to a measurable drop in performance on the corresponding class set, confirming that each extractor contributes to knowledge retention.
• Fine-tuning old feature extractors degraded performance, proving that the gains come from preserved knowledge in fixed extractors, not increased capacity.
• A simple distillation strategy using multiple past models performed worse than the proposed fusion mechanism, demonstrating the superiority of feature fusion over direct distillation.