[Paper Review] Overcoming Multi-Model Forgetting

Yassine Benyahia, Kaicheng Yu+5 more|ArXiv.org|Feb 21, 2019

Domain Adaptation and Few-Shot Learning12 citations

TL;DR

This paper introduces Weight Plasticity Loss (WPL), a statistically motivated regularization method that reduces multi-model forgetting—performance degradation in previously trained models due to shared parameter updates during sequential training of deep networks. WPL dynamically adjusts learning rates for shared weights based on their importance to prior models, reducing forgetting by up to 99% in strict convergence and 52% in loose convergence settings, with significant improvements in neural architecture search (e.g., 61.9 perplexity on PTB vs. 65.01 without WPL).

ABSTRACT

We identify a phenomenon, which we refer to as multi-model forgetting, that occurs when sequentially training multiple deep networks with partially-shared parameters; the performance of previously-trained models degrades as one optimizes a subsequent one, due to the overwriting of shared parameters. To overcome this, we introduce a statistically-justified weight plasticity loss that regularizes the learning of a model's shared parameters according to their importance for the previous models, and demonstrate its effectiveness when training two models sequentially and for neural architecture search. Adding weight plasticity in neural architecture search preserves the best models to the end of the search and yields improved results in both natural language processing and computer vision tasks.

Motivation & Objective

• Address the problem of multi-model forgetting, where training subsequent models degrades performance of previously trained models due to overwriting of shared parameters.
• Identify that performance degradation increases with the number of shared layers, especially in sequential training with parameter sharing.
• Propose a statistically grounded regularization method that preserves previously learned knowledge by adjusting learning dynamics based on parameter importance.
• Demonstrate the effectiveness of WPL in both strict and loose convergence scenarios, particularly in neural architecture search (NAS) pipelines.
• Show that reducing forgetting leads to better final model performance in NLP and computer vision tasks without extensive hyperparameter tuning.

Proposed method

• Formulate the learning objective as maximizing the posterior probability of shared and private parameters given training data, under mild assumptions.
• Derive a Weight Plasticity Loss (WPL) that regularizes updates to shared parameters by making them inversely proportional to their importance for previously trained models.
• Estimate parameter importance using Fisher information, enabling the model to slow down updates to critical weights for earlier models.
• Apply WPL in a sequential training framework where models are trained one after another, with shared layers updated during each phase.
• Integrate WPL into neural architecture search (NAS) methods like ENAS and NAO, where weight sharing is used to accelerate search.
• Use output dropout and path dropping strategies in NAS experiments to evaluate robustness and generalization of WPL across different search configurations.

Experimental results

Research questions

• RQ1 To what extent does multi-model forgetting degrade performance in sequentially trained models with shared parameters?
• RQ2 Can a statistically principled regularization method reduce forgetting while maintaining training efficiency in shared-parameter settings?
• RQ3 How effective is WPL in reducing forgetting during neural architecture search, where models are trained in a loose convergence regime?
• RQ4 Does WPL improve final model accuracy in downstream tasks such as language modeling and image classification compared to standard NAS methods?
• RQ5 Can WPL generalize across different NAS frameworks, such as ENAS and NAO, under varying dropout and weight-sharing strategies?

Key findings

• WPL reduces multi-model forgetting by up to 99% when model A is trained to full convergence, significantly preserving its performance during subsequent training of model B.
• In the more realistic loose convergence setting, WPL reduces forgetting by 52%, demonstrating robustness in practical training scenarios.
• In neural architecture search using ENAS, WPL reduces forgetting by 51% on the most affected model and 95% on average across all sampled models.
• The final best architecture found with WPL achieves a perplexity of 61.9 on the PTB language modeling dataset, outperforming ENAS (65.01) and even surpassing ENAS* (63.26) without extensive hyperparameter tuning.
• For image classification on CIFAR10, WPL reduces top-1 error from 4.87% (ENAS) to 3.81%, showing consistent gains across vision and NLP tasks.
• WPL generalizes effectively to other NAS methods such as NAO, reducing forgetting across different dropout rates and training strategies, including those with aggressive path dropping.