[Paper Review] MergePipe: A Budget-Aware Parameter Management System for Scalable LLM Merging

Yuanyi Wang, Yanggan Gu+7 more|arXiv (Cornell University)|Feb 5, 2026

Scientific Computing and Data Management0 citations

TL;DR

MergePipe treats LLM merging as a data management problem, introducing budget-aware planning and a catalog-driven execution engine to bound expert parameter reads, achieving significant I/O and wall-time reductions compared to naive pipelines.

ABSTRACT

Large language model (LLM) merging has become a key technique in modern LLM development pipelines, enabling the integration of multiple task- or domain-specific expert models without retraining. However, as the number of experts grows, existing merging implementations treat model parameters as unstructured files and execute merges in a stateless, one-shot manner, leading to excessive disk I/O, redundant parameter scans, and poor scalability. In this paper, we present extbf{MergePipe}, a parameter management system for scalable LLM merging. MergePipe is the first system that treats LLM merging as a data management and execution problem, and introduces a catalog-driven abstraction over model parameters, merge plans, and execution lineage. At its core, MergePipe employs a cost-aware planner that explicitly models expert parameter I/O and enforces user-specified I/O budgets, followed by a streaming execution engine that materializes merged models under transactional guarantees. Our key insight is that while base model reads and output writes are unavoidable, expert parameter reads dominate merge cost and constitute the primary optimization target. By making expert access budget-aware throughout planning and execution, MergePipe mitigates the $O(K)$ I/O growth of naive pipelines and achieves predictable scaling behavior. Experiments show that MergePipe reduces total I/O by up to an order of magnitude and delivers up to $11 imes$ end-to-end speedups (up to 90\% wall-time reduction) over state-of-the-art LLM merging pipelines.

Motivation & Objective

• Motivate scalable LLM merging by identifying expert-read I/O as the dominant cost in naive pipelines.
• Propose a catalog-driven, budget-aware system that decouples storage, planning, and execution for iterative merging.
• Enable reuse, provenance, and transactional materialization of merged checkpoints under user-specified I/O budgets.

Proposed method

• Introduce a persistent catalog to model parameter blocks, merge plans, and execution lineage as first-class data objects.
• Develop a cost model that isolates base/read/write costs from expert-read costs, and expresses expert I/O as a budgeted resource.
• Design a budget-aware planner that selects parameter blocks under a user-specified expert I/O budget, guided by conflict-aware signals from catalog metadata.
• Implement a streaming execution engine (DeltaIterator) that materializes merged checkpoints under transactional guarantees and exact plan adherence.
• Provide atomic publish and manifest-based lineage for reproducible and auditable merges.

Figure 1 . Naive LLM merging vs. MergePipe. Naive pipelines merge in a stateless, one-shot manner and repeatedly scans expert checkpoints ( $O(K)$ expert I/O), while MergePipe introduces planning, reuse, and budget-bounded expert reads.

Experimental results

Research questions

• RQ1 Can expert parameter reads be modeled as a budgeted resource to bound I/O growth in LLM merging?
• RQ2 Does budget-aware planning and execution enable predictable scaling across diverse merge operators and model sizes?
• RQ3 What are the trade-offs between planning overhead, reuse, and execution cost in iterative merging workflows?

Key findings

• Expert reads dominate merge cost and grow linearly with the number of experts in naive pipelines.
• Enforcing an explicit expert I/O budget keeps expert reads nearly flat as the number of experts increases, reducing wall time.
• Total I/O shifts from expert reads toward base reads and output writes as budgets constrain expert access.
• MergePipe achieves up to an order of magnitude reduction in total I/O and up to 11x end-to-end speedups (up to 90% wall-time reduction) over state-of-the-art pipelines.
• Planning, metadata management, and transactional execution incur small overheads that are amortized in iterative merging workflows.

Figure 2 . Naive merging scales poorly with experts. For 3B models, both TIES and DARE show near-linear growth in total I/O (bars) as $K$ increases, and wall time rises accordingly.