[Paper Review] Configurable Foundation Models: Building LLMs from a Modular Perspective

Chaojun Xiao, Zhengyan Zhang+21 more|arXiv (Cornell University)|Sep 4, 2024

Semantic Web and Ontologies4 citations

TL;DR

This paper introduces configurable foundation models (CFMs) as a modular framework for building large language models (LLMs) using 'bricks'—functional neuron partitions that emerge during pre-training or are customized post-training. By enabling dynamic operations like routing, merging, updating, and growing, CFMs achieve computational efficiency, scalability, and adaptability, with empirical evidence showing functional specialization in FFN layers of Llama-3-8B and Mistral-7B models.

ABSTRACT

Advancements in LLMs have recently unveiled challenges tied to computational efficiency and continual scalability due to their requirements of huge parameters, making the applications and evolution of these models on devices with limited computation resources and scenarios requiring various abilities increasingly cumbersome. Inspired by modularity within the human brain, there is a growing tendency to decompose LLMs into numerous functional modules, allowing for inference with part of modules and dynamic assembly of modules to tackle complex tasks, such as mixture-of-experts. To highlight the inherent efficiency and composability of the modular approach, we coin the term brick to represent each functional module, designating the modularized structure as configurable foundation models. In this paper, we offer a comprehensive overview and investigation of the construction, utilization, and limitation of configurable foundation models. We first formalize modules into emergent bricks - functional neuron partitions that emerge during the pre-training phase, and customized bricks - bricks constructed via additional post-training to improve the capabilities and knowledge of LLMs. Based on diverse functional bricks, we further present four brick-oriented operations: retrieval and routing, merging, updating, and growing. These operations allow for dynamic configuration of LLMs based on instructions to handle complex tasks. To verify our perspective, we conduct an empirical analysis on widely-used LLMs. We find that the FFN layers follow modular patterns with functional specialization of neurons and functional neuron partitions. Finally, we highlight several open issues and directions for future research. Overall, this paper aims to offer a fresh modular perspective on existing LLM research and inspire the future creation of more efficient and scalable foundational models.

Motivation & Objective

• Address the computational inefficiency and scalability challenges of monolithic large language models (LLMs) in resource-constrained and multi-domain environments.
• Overcome the limitations of static, full-parameter LLMs in deploying on edge devices and adapting to evolving, diverse application scenarios.
• Enable efficient, dynamic, and composable LLMs through a modular architecture based on functional neuron partitions (bricks) and configurable operations.
• Explore the feasibility and benefits of decomposing LLMs into reusable, traceable, and updatable functional components for sustainable model evolution.
• Lay the foundation for future research on scalable, efficient, and interoperable AI systems through a systematic modular perspective on foundation models.

Proposed method

• Introduce 'bricks' as functional modules: emergent bricks arise from parameter differentiation during pre-training, while customized bricks are created via post-training to enhance specific capabilities.
• Define four core operations: (1) routing and retrieval of relevant bricks based on instructions, (2) merging bricks via parameter averaging or stitching for composite tasks, (3) updating bricks through fine-tuning or injection, and (4) growing bricks during pre- or post-training to expand model capacity.
• Formalize brick granularity across five levels: solitary neuron, neuron group, layer, full model, and hybrid, enabling flexible configuration based on task needs.
• Conduct empirical analysis on Llama-3-8B-Instruct and Mistral-7B-Instruct-v0.3, measuring sparse activation, functionality localization, and specialization in feed-forward network (FFN) layers.
• Use intrinsic dimension analysis to assess the representational capacity of LLMs and validate the existence of functional neuron partitions.
• Propose a framework for multi-model cooperation using model-level bricks, with potential for unified intermediate representations or intermediary bridging models to enable efficient cross-model communication.

Figure 1: The illustration of a configurable foundation model consisting of emergent and customized bricks. For a given instruction, we select and combine tiny bricks to build an efficient instruction-specific model with minimal performance loss.

Experimental results

Research questions

• RQ1 Can functional specialization be observed in the neuron-level parameters of pre-trained LLMs, particularly in FFN layers?
• RQ2 How do emergent bricks—functionally specialized neuron partitions—arise during pre-training, and what is their relationship to downstream capabilities?
• RQ3 To what extent can customized bricks be effectively injected and updated to enhance model capabilities without catastrophic forgetting?
• RQ4 What are the most effective and efficient operations (e.g., routing, merging, growing) for dynamically composing modular LLMs to handle complex, diverse instructions?
• RQ5 How can scalable, interoperable, and communication-efficient multi-model systems be designed using model-level bricks as components?

Key findings

• Empirical analysis confirms that FFN layers in Llama-3-8B-Instruct and Mistral-7B-Instruct-v0.3 exhibit strong functional specialization, with neurons clustering into distinct functional partitions.
• Sparse activation patterns in FFN layers indicate that only a subset of neurons are actively engaged per input, supporting the modularity hypothesis.
• Emergent bricks demonstrate functional localization, where specific neurons consistently activate for specific tasks or concepts, suggesting inherent modularity in pre-trained LLMs.
• Customized bricks—such as task-specific, knowledge-augmented, or modality-specific modules—can be effectively created via post-training and integrated into the model.
• The proposed operations (routing, merging, updating, growing) enable dynamic configuration of LLMs, allowing them to adapt to complex instructions by assembling only relevant bricks.
• The modular framework enables computational efficiency, parameter reusability, and traceability, with potential for distributed and scalable inference and continual learning.

Figure 2: The illustration for emergent bricks. In the randomly initialized, functional differentiation emerges following the pre-training phase. Neurons with similar functionalities can be aggregated to form small functional bricks. Here, the model is divided into two human-defined layer bricks, of