[Paper Review] A Combinatorial Perspective on Transfer Learning

Jianan Wang, Eren Sezener+3 more|arXiv (Cornell University)|Oct 23, 2020

Domain Adaptation and Few-Shot Learning35 references4 citations

TL;DR

This paper proposes a combinatorial transfer learning framework that enables neural networks to generalize to exponentially more unseen tasks by locally ensembling modular, task-specific nodes. Using the Forget-Me-Not Process for task segmentation and memory-based ensembling, and Gated Linear Networks for modular, local learning, the method achieves robust continual learning with no catastrophic forgetting and increasing positive transfer, outperforming state-of-the-art online and offline methods on Split/Permuted MNIST and real-world electricity market data.

ABSTRACT

Human intelligence is characterized not only by the capacity to learn complex skills, but the ability to rapidly adapt and acquire new skills within an ever-changing environment. In this work we study how the learning of modular solutions can allow for effective generalization to both unseen and potentially differently distributed data. Our main postulate is that the combination of task segmentation, modular learning and memory-based ensembling can give rise to generalization on an exponentially growing number of unseen tasks. We provide a concrete instantiation of this idea using a combination of: (1) the Forget-Me-Not Process, for task segmentation and memory based ensembling; and (2) Gated Linear Networks, which in contrast to contemporary deep learning techniques use a modular and local learning mechanism. We demonstrate that this system exhibits a number of desirable continual learning properties: robustness to catastrophic forgetting, no negative transfer and increasing levels of positive transfer as more tasks are seen. We show competitive performance against both offline and online methods on standard continual learning benchmarks.

Motivation & Objective

• To address the limitations of deep learning in online continual learning, where models suffer from catastrophic forgetting and lack data efficiency.
• To enable generalization to an exponentially growing number of unseen tasks through modular, combinatorial transfer.
• To develop a framework that leverages task segmentation, modular learning, and memory-based ensembling without requiring task boundaries or identities.
• To demonstrate robust positive forward and backward transfer in online settings with no negative transfer.
• To provide a data-efficient, privacy-friendly alternative to batch i.i.d. training by using online, streaming data.

Proposed method

• The framework uses the Forget-Me-Not Process to dynamically segment tasks and maintain a memory of past task-specific models.
• It employs Gated Linear Networks (GLNs) to enable modular, local learning per neuron, supporting explicit node-level ensembling.
• Task-specific networks are locally ensembled at the node level, creating $ h^m $ possible pseudo-task configurations from $ h $ tasks and $ m $ nodes.
• The system uses geometric mixing for probabilistic prediction and online weight updates, enabling continual adaptation without storing raw data.
• The method avoids reliance on batch i.i.d. training and does not require explicit task identity or boundary detection.
• It combines GLNs and FMN to achieve a complementary system that supports combinatorial transfer and robustness to distributional drift.

Experimental results

Research questions

• RQ1 Can a neural network trained on a sequence of tasks generalize to an exponentially larger number of unseen pseudo-tasks through local node ensembling?
• RQ2 Can modular, local learning mechanisms like GLNs enable effective combinatorial transfer without catastrophic forgetting in online continual learning?
• RQ3 Does the system exhibit positive forward and backward transfer without requiring task boundaries or identities?
• RQ4 How does the method perform under data drift when task identities are unknown, compared to existing continual learning algorithms?
• RQ5 Can the framework maintain high performance on real-world, non-stationary data streams without storing past data?

Key findings

• The proposed nctl algorithm achieved 97.31 ± 0.11 accuracy on the Split Fashion MNIST benchmark, outperforming prior online methods and approaching the offline upper bound of 98.59 ± 0.15.
• On the Electricity (Elec2-3) dataset with unknown distributional drift, nctl achieved 86.37% accuracy, significantly outperforming SPLICE-2 (66–67.7%) and DWM-NB (80.75%).
• The system demonstrated no catastrophic forgetting and increasing positive backward transfer as more tasks were learned, indicating effective knowledge retention.
• Combinatorial transfer was empirically validated: the model generalized to $ h^m $ pseudo-tasks, with semantically meaningful task combinations.
• The method showed robustness to distributional shifts and did not require task identity or boundary detection, enabling true online continual learning.
• The framework achieved competitive performance without storing raw data, supporting privacy-preserving learning.