[Paper Review] Expert Gate: Lifelong Learning with a Network of Experts

Rahaf Aljundi, Punarjay Chakravarty+1 more|arXiv (Cornell University)|Nov 18, 2016

Domain Adaptation and Few-Shot Learning4 citations

TL;DR

This paper proposes Expert Gate, a lifelong learning framework that dynamically selects the most relevant expert model at test time using gating autoencoders, enabling efficient, scalable, and memory-efficient inference without storing past data. It outperforms both sequential fine-tuning and joint training on image and video prediction tasks by leveraging task-relatedness for selective knowledge transfer and automatic expert routing.

ABSTRACT

In this paper we introduce a model of lifelong learning, based on a Network of Experts. New tasks / experts are learned and added to the model sequentially, building on what was learned before. To ensure scalability of this process,data from previous tasks cannot be stored and hence is not available when learning a new task. A critical issue in such context, not addressed in the literature so far, relates to the decision which expert to deploy at test time. We introduce a set of gating autoencoders that learn a representation for the task at hand, and, at test time, automatically forward the test sample to the relevant expert. This also brings memory efficiency as only one expert network has to be loaded into memory at any given time. Further, the autoencoders inherently capture the relatedness of one task to another, based on which the most relevant prior model to be used for training a new expert, with finetuning or learning without-forgetting, can be selected. We evaluate our method on image classification and video prediction problems.

Motivation & Objective

• To address the challenge of selecting the most relevant expert model at test time in lifelong learning without storing previous training data.
• To enable scalable lifelong learning by dynamically routing inputs to the most appropriate expert network based on input content.
• To reduce memory overhead by loading only one expert model at a time, using a lightweight gating mechanism.
• To improve knowledge transfer by identifying the most related prior experts for fine-tuning or learning-without-forgetting.
• To demonstrate the effectiveness of autoencoders as a task recognizer that matches discriminative classifiers in performance without requiring labeled data.

Proposed method

• A Network of Experts is constructed, where each expert is a task-specific neural network trained sequentially on new data.
• Gating autoencoders are trained per expert to learn a compact, shared representation of the task's input distribution.
• At test time, the autoencoder for each expert evaluates the input and routes it to the most relevant expert based on similarity to the training distribution.
• The autoencoders are trained end-to-end with their corresponding expert networks to ensure task-aware routing.
• The system uses task-relatedness scores derived from autoencoder reconstructions to guide knowledge transfer during training of new experts.
• The method avoids catastrophic forgetting by selectively fine-tuning only the most relevant prior experts, rather than all previous models.

Experimental results

Research questions

• RQ1 Can a data-free gating mechanism effectively route test samples to the most relevant expert model in a lifelong learning setup?
• RQ2 How does the performance of autoencoder-based gating compare to discriminative classifiers trained on full data for task recognition?
• RQ3 Can autoencoders be used to identify the most related prior experts for knowledge transfer, improving learning-without-forgetting?
• RQ4 Does Expert Gate outperform sequential fine-tuning and joint training in terms of accuracy and memory efficiency?
• RQ5 Can the gating mechanism generalize across diverse tasks such as image classification and video prediction?

Key findings

• Expert Gate achieved an average L1 distance of 23.4 on video prediction, outperforming sequential fine-tuning (31.1) and matching joint training (23.8), despite not requiring access to all past data.
• On image classification, Expert Gate matched or exceeded the performance of joint training while avoiding catastrophic forgetting and reducing memory load.
• The gating autoencoders achieved task recognition accuracy comparable to a discriminative classifier trained on all data, demonstrating their effectiveness as a data-free task recognizer.
• The system successfully reduced memory usage by loading only one expert model at inference time, enabling scalability on resource-constrained devices.
• Qualitative results showed that Expert Gate produced more accurate video predictions, particularly in preserving lane markers and scene structure, compared to sequential fine-tuning.
• The autoencoders effectively captured task relatedness, enabling informed selection of prior experts for knowledge transfer, which improved generalization and reduced forgetting.