[論文レビュー] Modular Deep Learning

Jonas Pfeiffer, Sebastian Ruder+2 more|arXiv (Cornell University)|Feb 22, 2023

Ferroelectric and Negative Capacitance Devices被引用数 12

ひとこと要約

モジュラー深層学習アーキテクチャの調査。モジュール、ルーティング、集約、訓練コンポーネントがどのように相互作用して、NLP、ビジョン、音声、RLタスク全体で正の転移、構成的一般化、パラメータ効率を可能にするかを詳述する。

ABSTRACT

Transfer learning has recently become the dominant paradigm of machine learning. Pre-trained models fine-tuned for downstream tasks achieve better performance with fewer labelled examples. Nonetheless, it remains unclear how to develop models that specialise towards multiple tasks without incurring negative interference and that generalise systematically to non-identically distributed tasks. Modular deep learning has emerged as a promising solution to these challenges. In this framework, units of computation are often implemented as autonomous parameter-efficient modules. Information is conditionally routed to a subset of modules and subsequently aggregated. These properties enable positive transfer and systematic generalisation by separating computation from routing and updating modules locally. We offer a survey of modular architectures, providing a unified view over several threads of research that evolved independently in the scientific literature. Moreover, we explore various additional purposes of modularity, including scaling language models, causal inference, programme induction, and planning in reinforcement learning. Finally, we report various concrete applications where modularity has been successfully deployed such as cross-lingual and cross-modal knowledge transfer. Related talks and projects to this survey, are available at https://www.modulardeeplearning.com/.

研究の動機と目的

• Explain why modularity helps with transfer learning and systematic generalisation.
• Provide a unified taxonomy of modular deep learning, covering computation, routing, aggregation, and training.
• Survey existing module implementations, routing strategies, and aggregation mechanisms and how they interact with training settings.
• Highlight applications and future directions for modularity in NLP, vision, speech, and reinforcement learning.

提案手法

• Define a general modular function composed of computation, routing, and aggregation blocks.
• Categorise computation into parameter, input, and function composition plus hypernetworks.
• Differentiate routing by fixed versus learned, and soft versus hard routing, including top-k and Mixture-of-Experts variants.
• Describe aggregation strategies ranging from deterministic weighting to attention-based learnable aggregators.
• Discuss training settings: joint multitask learning, continual learning, and post-hoc modularisation of pre-trained models.
• Provide a unified notation and algorithmic view of modular function forward passes (Algorithm 1).

実験結果

リサーチクエスチョン

• RQ1 How can modular architectures mitigate interference and forgetting in multi-task and continual learning?
• RQ2 What are the design choices (computation, routing, aggregation) that enable positive transfer and systematic generalisation?
• RQ3 How do various modular approaches scale across languages, modalities, and tasks?
• RQ4 What are practical training regimes to deploy modular components with pre-trained backbones?
• RQ5 Which applications demonstrate the benefits of modularity in transfer learning and generalisation?

主な発見

• Modularity decouples computation from routing and updating, enabling local module updates and improved robustness to distribution shifts.
• A unified view shows many methods as combinations of computation, routing, aggregation, and training, clarifying connections across literature.
• Sparse and low-rank adapters, prompt-based, and hypernetwork-based modules offer parameter-efficient fine-tuning and scalable growth.
• Hard routing with selective module activation enables conditional computation and modular specialisation, while soft routing provides gradient-friendly training.
• Modularity supports cross-lingual, cross-modal knowledge transfer and broader transfer learning applications across NLP, CV, and speech.
• Modularity also informs hierarchical RL, programme simulation, causal discovery, and general-purpose agents.