[论文解读] RLHFless: Serverless Computing for Efficient RLHF

Rui Wei, Hanfei Yu+6 more|arXiv (Cornell University)|Feb 26, 2026

Explainable Artificial Intelligence (XAI)被引用 0

一句话总结

tldr: RLHFless 是基于无服务器计算的同步 RLHF 的首个可扩展训练框架，实现动态资源自适应、共享前缀的预计算，以及面向成本的参与者扩缩以减少空闲时间和成本。

ABSTRACT

Reinforcement Learning from Human Feedback (RLHF) has been widely applied to Large Language Model (LLM) post-training to align model outputs with human preferences. Recent models, such as DeepSeek-R1, have also shown RLHF's potential to improve LLM reasoning on complex tasks. In RL, inference and training co-exist, creating dynamic resource demands throughout the workflow. Compared to traditional RL, RLHF further challenges training efficiency due to expanding model sizes and resource consumption. Several RLHF frameworks aim to balance flexible abstraction and efficient execution. However, they rely on serverful infrastructures, which struggle with fine-grained resource variability. As a result, during synchronous RLHF training, idle time between or within RL components often causes overhead and resource wastage. To address these issues, we present RLHFless, the first scalable training framework for synchronous RLHF, built on serverless computing environments. RLHFless adapts to dynamic resource demands throughout the RLHF pipeline, pre-computes shared prefixes to avoid repeated computation, and uses a cost-aware actor scaling strategy that accounts for response length variation to find sweet spots with lower cost and higher speed. In addition, RLHFless assigns workloads efficiently to reduce intra-function imbalance and idle time. Experiments on both physical testbeds and a large-scale simulated cluster show that RLHFless achieves up to 1.35x speedup and 44.8% cost reduction compared to the state-of-the-art baseline.

研究动机与目标

• Motivate improving RLHF training efficiency amid growing model sizes and resource demands.
• Address inefficiencies from idle time and resource variability in serverful RLHF infrastructures.
• Introduce a scalable serverless framework that adapts to dynamic RLHF workloads.
• Reduce redundant computation and balance workloads to minimize idle time and imbalance.

提出的方法

• Adapt RLHF pipeline to serverless environments to handle dynamic resource demands.
• Pre-compute shared prefixes to avoid repeated computation in RLHF workflows.
• Implement a cost-aware actor scaling strategy that accounts for response length variation.
• Efficient workload assignment to reduce intra-function imbalance and idle time.
• Evaluate RLHFless on physical testbeds and large-scale simulated clusters.

实验结果

研究问题

• RQ1 How can serverless computing be leveraged to efficiently run synchronous RLHF training?
• RQ2 What techniques (e.g., pre-computation, cost-aware scaling) reduce idle time and cost in RLHF workflows?
• RQ3 How does RLHFless perform in terms of speed and cost compared with state-of-the-art baselines under varying resource conditions?
• RQ4 What workload management strategies minimize intra-function imbalance in RLHF pipelines?

主要发现

• RLHFless achieves up to 1.35x speedup over the baseline.
• RLHFless reduces cost by up to 44.8% compared to the baseline.
• The framework adapts to dynamic resource demands and reduces idle time through pre-computation and workload balancing.
• Experiments on physical testbeds and a large-scale simulated cluster validate efficiency gains.