[论文解读] MergePipe: A Budget-Aware Parameter Management System for Scalable LLM Merging

Yuanyi Wang, Yanggan Gu+7 more|arXiv (Cornell University)|Feb 5, 2026

Scientific Computing and Data Management被引用 0

一句话总结

MergePipe将LLM合并视为数据管理问题，引入基于预算的规划和目录驱动的执行引擎，以限制专家参数读取，显著降低I/O和 Wall-time，相较于天真管线。

ABSTRACT

Large language model (LLM) merging has become a key technique in modern LLM development pipelines, enabling the integration of multiple task- or domain-specific expert models without retraining. However, as the number of experts grows, existing merging implementations treat model parameters as unstructured files and execute merges in a stateless, one-shot manner, leading to excessive disk I/O, redundant parameter scans, and poor scalability. In this paper, we present extbf{MergePipe}, a parameter management system for scalable LLM merging. MergePipe is the first system that treats LLM merging as a data management and execution problem, and introduces a catalog-driven abstraction over model parameters, merge plans, and execution lineage. At its core, MergePipe employs a cost-aware planner that explicitly models expert parameter I/O and enforces user-specified I/O budgets, followed by a streaming execution engine that materializes merged models under transactional guarantees. Our key insight is that while base model reads and output writes are unavoidable, expert parameter reads dominate merge cost and constitute the primary optimization target. By making expert access budget-aware throughout planning and execution, MergePipe mitigates the $O(K)$ I/O growth of naive pipelines and achieves predictable scaling behavior. Experiments show that MergePipe reduces total I/O by up to an order of magnitude and delivers up to $11 imes$ end-to-end speedups (up to 90\% wall-time reduction) over state-of-the-art LLM merging pipelines.

研究动机与目标

• 通过识别 naive 管道中专家读取I/O 为主导成本，动机化可扩展的LLM合并。
• 提出一个目录驱动、预算感知的系统，将存储、规划与执行解耦，以实现迭代合并。
• 在用户指定的I/O预算下实现合并检查点的复用、溯源和事务性物化。

提出的方法

• 引入一个持久目录，将参数块、合并计划和执行血缘关系建模为一等数据对象。
• 开发一个成本模型，将基础/读取/写入成本与专家读取成本分离，并将专家I/O表示为一个预算资源。
• 设计一个预算感知的规划器，在用户指定的专家I/O预算下选择参数块，并以目录元数据中的冲突感知信号为引导。
• 实现一个流式执行引擎（DeltaIterator），在事务保证和严格遵循计划的前提下对合并检查点进行物化。
• 提供原子发布和基于清单的血缘，以实现可重复和可审计的合并。

Figure 1 . Naive LLM merging vs. MergePipe. Naive pipelines merge in a stateless, one-shot manner and repeatedly scans expert checkpoints ( $O(K)$ expert I/O), while MergePipe introduces planning, reuse, and budget-bounded expert reads.

实验结果

研究问题

• RQ1 专家参数读取是否可以建模为预算资源，以限制LLM合并中的I/O增长？
• RQ2 预算感知的规划与执行是否能够在不同合并算子和模型规模中实现可预测的扩展？
• RQ3 在迭代合并工作流中，规划开销、复用和执行成本之间的权衡是什么？

主要发现

• 专家读取主导合并成本，并随 naive 管道中的专家数量线性增长。
• 强制执行显式的专家I/O预算使专家读取在专家数量增加时几乎保持平坦，降低了墙钟时间。
• 随着预算约束专家访问，Total I/O 从专家读取转向基础读取和输出写入。
• MergePipe在总I/O方面实现了最多一个数量级的降低，在端到端速度方面实现最多11x的提升（墙钟时间最多降低90%），相对于最先进的管线。
• 规划、元数据管理和事务执行在迭代合并工作流中产生的小开销是可摊销的。

Figure 2 . Naive merging scales poorly with experts. For 3B models, both TIES and DARE show near-linear growth in total I/O (bars) as $K$ increases, and wall time rises accordingly.