[Paper Review] Simple Lifelong Learning Machines

Jayanta Dey, Joshua T. Vogelstein+14 more|arXiv (Cornell University)|Apr 27, 2020

Computability, Logic, AI Algorithms4 citations

TL;DR

This paper proposes Synergistic Forests (SynF) and Synergistic Networks (SynN), lifelong learning algorithms that ensemble representations from independently trained models on different tasks to enable both forward and backward transfer. By leveraging quasilinear-time ensembling of task-specific representations, the methods achieve significant performance gains across vision, audition, and tabular benchmarks, outperforming existing approaches in both low- and high-data regimes.

ABSTRACT

In lifelong learning, data are used to improve performance not only on the present task, but also on past and future (unencountered) tasks. While typical transfer learning algorithms can improve performance on future tasks, their performance on prior tasks degrades upon learning new tasks (called forgetting). Many recent approaches for continual or lifelong learning have attempted to maintain performance on old tasks given new tasks. But striving to avoid forgetting sets the goal unnecessarily low. The goal of lifelong learning should be to use data to improve performance on both future tasks (forward transfer) and past tasks (backward transfer). In this paper, we show that a simple approach -- representation ensembling -- demonstrates both forward and backward transfer in a variety of simulated and benchmark data scenarios, including tabular, vision (CIFAR-100, 5-dataset, Split Mini-Imagenet, and Food1k), and speech (spoken digit), in contrast to various reference algorithms, which typically failed to transfer either forward or backward, or both. Moreover, our proposed approach can flexibly operate with or without a computational budget.

Motivation & Objective

• To address the limitations of existing lifelong learning algorithms that fail to achieve both forward and backward transfer.
• To develop a scalable, quasilinear-time method that enables continual improvement on both past and future tasks through representation ensembling.
• To demonstrate that forward and backward transfer can be systematically achieved in low-data regimes, where transfer is most critical.
• To provide a resource-growing approach that dynamically incorporates new representations without catastrophic forgetting.
• To validate the method across diverse benchmarks, including vision (CIFAR-100, Food1k, Split Mini-Imagenet), audition (spoken digits), and tabular data.

Proposed method

• The method ensembles representations learned independently on each task using a bagging-style approach, where each task is modeled with a separate estimator (random forest or deep neural network).
• For each new task, the system stores the trained representation and later combines it with existing representations via averaging or voting to form a unified predictor.
• The inference process uses all previously learned representations as features, enabling both forward and backward transfer without retraining.
• The computational complexity is quasilinear in the number of tasks and samples, achieved by avoiding full retraining or parameter regularization.
• The approach is task-aware, meaning the system knows which task a sample belongs to during inference, enabling selective representation fusion.
• Ablation studies show that performance improves with increased representation capacity and with replay of old task data.

Experimental results

Research questions

• RQ1 Can representation ensembling across independently trained models enable both forward and backward transfer in lifelong learning?
• RQ2 Does the proposed method achieve significant transfer in low-data regimes, where transfer is most needed?
• RQ3 How does the performance of the representation ensembling approach compare to existing lifelong learning algorithms across diverse benchmarks?
• RQ4 What is the impact of representation capacity and replay of old data on the performance of the ensembled system?
• RQ5 Can quasilinear-time algorithms effectively scale to large-scale lifelong learning scenarios with many tasks and high-dimensional data?

Key findings

• SynF and SynN outperform all reference algorithms in terms of forward, backward, and overall transfer across vision, audition, and tabular datasets.
• On the spoken digit dataset, SynF and SynN achieve positive backward and forward transfer, while other methods fail to achieve both simultaneously.
• On the Food1k 50X20 dataset, SynN achieves the best performance among all algorithms, demonstrating strong scalability and transfer in large-scale settings.
• On Split Mini-Imagenet and 5-dataset, SynF and SynN show less forgetting than reference methods, even with relatively low sample sizes per task.
• Ablation studies confirm that increasing representation capacity and replaying old task data significantly improve performance, validating the importance of these components.
• The results show that bagging-based lifelong learning (SynF/SynN) outperforms boosting-based approaches (e.g., Model Zoo) in low-data regimes, consistent with known behavior in single-task learning.