[Paper Review] Less-forgetful Learning for Domain Expansion in Deep Neural Networks

Heechul Jung, Jeongwoo Ju+2 more|arXiv (Cornell University)|Nov 16, 2017

Domain Adaptation and Few-Shot Learning25 references3 citations

TL;DR

This paper proposes a less-forgetful learning (LF) method for deep neural networks to enable domain expansion—retaining performance on previously learned domains while adapting to new ones without access to old data. The approach uses two novel properties: feature-level knowledge distillation and an auxiliary loss ($\mathcal{L}_e$) to preserve old domain representations, achieving superior performance over fine-tuning, LwF, and EWC on image classification benchmarks.

ABSTRACT

Expanding the domain that deep neural network has already learned without accessing old domain data is a challenging task because deep neural networks forget previously learned information when learning new data from a new domain. In this paper, we propose a less-forgetful learning method for the domain expansion scenario. While existing domain adaptation techniques solely focused on adapting to new domains, the proposed technique focuses on working well with both old and new domains without needing to know whether the input is from the old or new domain. First, we present two naive approaches which will be problematic, then we provide a new method using two proposed properties for less-forgetful learning. Finally, we prove the effectiveness of our method through experiments on image classification tasks. All datasets used in the paper, will be released on our website for someone's follow-up study.

Motivation & Objective

• Address the challenge of catastrophic forgetting in deep neural networks when expanding to new domains without access to old training data.
• Enable continual learning in real-world applications where models must retain knowledge of previously learned domains while adapting to new ones.
• Develop a method that works effectively without requiring prior knowledge of whether input data belong to the old or new domain.
• Improve generalization and stability in domain expansion by preserving old domain features through novel loss functions and architectural constraints.

Proposed method

• Propose two key properties for less-forgetful learning: (1) feature-level knowledge distillation to preserve old domain representations, and (2) an auxiliary loss ($\mathcal{L}_e$) to stabilize feature space during training.
• Introduce a training objective combining cross-entropy loss with $\mathcal{L}_e$, which encourages feature consistency between old and new domain samples.
• Use a shared feature extractor with a classifier head trained on new domain data only, while preserving old domain performance via distillation.
• Apply the method in a fine-tuning setting where only new domain data are used, but old domain performance is maintained through the proposed loss and knowledge distillation.
• Design the network so that old domain features are well-mixed and indistinguishable, preventing catastrophic forgetting via $\mathcal{L}_e$.
• Validate the method on image classification tasks using datasets like MNIST, SVHN, CIFAR, and ImageNet, with ablation studies on scratch training and fine-tuning.

Experimental results

Research questions

• RQ1 Can a deep neural network retain performance on an old domain while being trained only on new domain data?
• RQ2 How can catastrophic forgetting be mitigated in domain expansion without access to old data or prior knowledge of input domain?
• RQ3 What loss functions and architectural components enable stable feature representation preservation across domains?
• RQ4 Does the proposed method generalize to continual learning scenarios involving sequential domain adaptation?
• RQ5 How does the performance of less-forgetful learning compare to fine-tuning, LwF, and EWC in terms of old and new domain accuracy?

Key findings

• The proposed less-forgetful (LF) learning method outperforms fine-tuning, LwF, and EWC in preserving performance on the old domain while adapting to the new domain.
• On the MNIST ∪ SVHN benchmark, LF achieved a classification rate of 71.1% in continual learning, significantly outperforming fine-tuning (67.18%) and approaching offline learning (78.16%).
• In the CIFAR Color ∪ Gray experiment, scratch training on new domain data (grayscale) achieved comparable performance on both domains, suggesting that complex data generalization improves cross-domain transfer.
• When new domain data are limited (e.g., ImageNet Normal ∪ Dark & Bright), scratch training led to overfitting and poor old domain performance, while LF maintained strong generalization.
• The $\mathcal{L}_e$ loss effectively stabilized the feature space, preventing significant changes and reducing forgetting, as evidenced by well-mixed old domain features.
• The method is feasible for continual learning: in a 10-task sequential CIFAR-10 experiment, LF achieved 71.1% accuracy, demonstrating robustness without access to previous data.