[论文解读] SLowRL: Safe Low-Rank Adaptation Reinforcement Learning for Locomotion

Elham Daneshmand, Shafeef Omar+3 more|arXiv (Cornell University)|Mar 17, 2026

Robotic Locomotion and Control被引用 0

一句话总结

TL;DR: SLowRL 结合 Low-Rank Adaptation (LoRA) 与在线安全恢复策略，在真实机器人上安全高效地微调仿真训练的运动策略，达到微调时间降低46.5%且安全违规几乎为零的效果。

ABSTRACT

Sim-to-real transfer of locomotion policies often leads to performance degradation due to the inevitable sim-to-real gap. Naively fine-tuning these policies directly on hardware is problematic, as it poses risks of mechanical failure and suffers from high sample inefficiency. In this paper, we address the challenge of safely and efficiently fine-tuning reinforcement learning (RL) policies for dynamic locomotion tasks. Specifically, we focus on fine-tuning policies learned in simulation directly on hardware, while explicitly enforcing safety constraints. In doing so, we introduce SLowRL, a framework that combines Low-Rank Adaptation (LoRA) with training-time safety enforcement via a recovery policy. We evaluate our method both in simulation and on a real Unitree Go2 quadruped robot for jump and trot tasks. Experimental results show that our method achieves a $46.5\%$ reduction in fine-tuning time and near-zero safety violations compared to standard proximal policy optimization (PPO) baselines. Notably, we find that a rank-1 adaptation alone is sufficient to recover pre-trained performance in the real world, while maintaining stable and safe real-world fine-tuning. These results demonstrate the practicality of safe, efficient fine-tuning for dynamic real-world robotic applications.

研究动机与目标

• 目标是以最小风险和样本成本将高保真仿真 locomotion 策略安全地适配到真实硬件上。
• 研究极低秩（如秩-1）更新是否足以实现有效的 sim-to-real 适配。
• 证明联合自编码器和评论者（actor 与 critic）的适配对于真实世界策略价值对齐的必要性。
• 展示将恢复安全策略整合进来能够在硬件故障极小的情况下实现更快的收敛。

提出的方法

• 从仿真中冻结预训练策略并并行学习低秩适配器（LoRA）以适应冻结权重。
• 在 actor 与 critic 的选定密集层注入 LoRA 适配器，产生一个加法 BAx 更新，输出在 ELU 激活前相加。
• 训练一个与任务无关的恢复策略，将机器人驱动回到安全的名义状态，由一个可覆盖主策略的安全筛选器触发。
• 使用秩-1 LoRA（rho=1）作为首选的适配维度，随机初始化 A，B=0 以从预训练行为开始。
• 证明同时适配 actor 与 critic 相较于只适配 actor 能更好地与真实世界动力学对齐。
• 表明在所有层上适配 LoRA 比仅在头部或部分层进行适配具有更好性能。

Figure 3 : The Unitree Go2 robot performs a dynamic jump. Using $SLowRL$ , we fine-tune a low-rank adapter on top of a frozen simulation-trained policy, achieving substantially faster training with near-zero safety violations compared to standard PPO .

实验结果

研究问题

• RQ1 极低秩（秩-1）通过 LoRA 的更新是否足以在不进行全面微调的情况下解决步态在仿真到现实的差距？
• RQ2 联合适配 actor 与 critic 是否对于稳定且有效的真实世界策略改进是必要的？
• RQ3 将学习到的恢复安全策略整合是否能够实现比标准 PPO 基线更快且更安全的在硬件上的微调？
• RQ4 应将 LoRA 插入网络的何处以最大化 sim-to-real 传输性能？
• RQ5 在真实世界适应过程中，安全筛选器加恢复策略的安全性和样本效率优势是什么？

主要发现

• SLowRL 相较于完整微调 PPO 基线，在真实世界微调收敛时间上实现了 46.5% 的减少。
• 与 FFT 基线不同，SLowRL 在 trot 与 jump 任务的种子间保持零硬件故障（跌落/碰撞）。
• 秩-1 LoRA 适配足以在真实世界恢复甚至提升预训练性能。
• 同时适配 actor 与 critic 对于使价值函数与真实世界动力学对齐是必要的，而仅适配 actor 无法收敛。
• 在所有层上适配 LoRA 使性能最好，能够对特征进行深度校正并实现鲁棒的 sim-to-real 传输。

Figure 4 : Sim-to-Sim Sample Efficiency during Fine-tuning (Trot Task) Comparative learning curves showing mean reward over wall-clock time for the trotting task. $SLowRL$ (blue) significantly outperforms the FFT and Zero-Shot baselines, achieving a $38\%$ reduction in time-to-convergence while main