[论文解读] From Robotics to Sepsis Treatment: Offline RL via Geometric Pessimism

Sarthak Wanjari|arXiv (Cornell University)|Feb 9, 2026

Reinforcement Learning in Robotics被引用 0

一句话总结

Geo-IQL 通过预计算的几何距离惩罚来惩罚 OOD 行为，从而增强隐式 Q 学习，在分裂数据上提高稳定性，并在机器人和败血症护理数据集上实现更安全、质量更高的策略。

ABSTRACT

Offline Reinforcement Learning (RL) promises the recovery of optimal policies from static datasets, yet it remains susceptible to the overestimation of out-of-distribution (OOD) actions, particularly in fractured and sparse data manifolds. Current solutions necessitate a trade-off between computational efficiency and performance. Methods like CQL offer rigorous conservatism but require tremendous compute power while efficient expectile-based methods like IQL often fail to correct OOD errors on pathological datasets, collapsing to Behavioural Cloning. In this work, we propose Geometric Pessimism, a modular, compute-efficient framework that augments standard IQL with density-based penalty derived from k-nearest-neighbour distances in the state-action embedding space. By pre-computing the penalties applied to each state-action pair, our method injects OOD conservatism via reward shaping with a O(1) training overhead to the training loop. Evaluated on the D4RL MuJoCo benchmark, our method, Geo-IQL outperforms standard IQL on sensitive and unstable medium-replay tasks by over 18 points, while reducing inter-seed standard-deviation by 4 times. Furthermore, Geo-IQL does not degrade performance on stable manifolds. Crucially, we validate our algorithm on the MIMIC-III Sepsis critical care dataset. While standard IQL collapses to behaviour cloning, Geo-IQL demonstrates active policy improvement. Maintaining safety constraints, it achieves 86.4% terminal agreement with clinicians compared to IQL's 75%. Our results suggest that geometric pessimism provides the necessary regularisation to safely overcome local optima in critical, real-world decision systems.

研究动机与目标

• 在数据分裂或稀疏的高风险领域推动更安全的离线 RL。
• 提出一种计算高效的方法，在样本内学习中引入几何惩罚。
• 实现惩罚的预计算以维持 O(1) 训练开销。
• 在机器人基准和危重护理数据上演示稳定性和策略质量的提升。

提出的方法

• 将状态-动作对嵌入到一个联合空间，并计算均值最近邻距离作为几何不确定性代理。
• 使用 MAD 以及安全核心阈值对距离进行鲁棒标准化，以创建分级风险表面 (U)。
• 对奖励应用密度自适应惩罚：r_geo(s,a)=r(s,a)−λ_adapt·max(0,U(s,a))。
• 在查找表中预计算惩罚，以在训练中实现 O(1) 的惩罚检索。
• 将惩罚与 IQL 的目标整合，使评估器保持不变，而评估器在惩罚奖励下学习的评估者。

Figure 1: 3-Dimensional visualisation of using geometry as a proxy for epistemic uncertainty.

实验结果

研究问题

• RQ1 几何距离到数据流形是否可以作为离线 RL 中认识上界不确定性的代理？
• RQ2 在有裂解的数据上加入预计算的几何惩罚是否能在不牺牲稳定性的前提下提升 IQL 性能？
• RQ3 Geo-IQL 在高风险现实世界数据集（如 MIMIC-III Sepsis）上相较于标准 IQL 和 CQL 的表现如何？
• RQ4 该方法的计算效率是否足以在普通硬件上运行？
• RQ5 在医疗保健领域，几何引导的离线 RL 能带来哪些安全性和临床对齐方面的好处？

主要发现

Task	BC	CQL	IQL	Geo-IQL
halfcheetah-medium-replay-v2	27.69 ± 10.92	45.41 ± 0.81	43.68 ± 4.15	42.52 ± 3.04
hopper-medium-replay-v2	51.87 ± 20.26	82.60 ± 21.10	80.09 ± 21.80	98.94 ± 5.33
walker2d-medium-replay-v2	43.17 ± 25.77	78.28 ± 18.85	80.17 ± 17.89	82.10 ± 13.39

• Geo-IQL 在 D4RL MuJoCo 套件中的 Hopper-medium-replay-v2 上比标准 IQL 高出 18 点以上。
• Geo-IQL 在敏感任务上的跨种子标准差降低约 4 倍。
• Geo-IQL 在稳定流形上与 IQL 表现相当，数据密集区域保持性能。
• 在 MIMIC-III Sepsis 上，Geo-IQL 实现更高的终端状态一致性（86.39% 对 75.02%）。
• Geo-IQL 实现了正向 Q 改善（ΔQ = +0.0138），而 IQL 显示负的 ΔQ（−0.0169）。
• Geo-IQL 在确保安全的前提下实现对关键护理中的策略改进。

Figure 2: Visualising the Adaptive Safety Mechanism. The blue cloud represents the training data manifold. Green Point: A query inside the dense region. The algorithm detects near-zero distance to neighbours and applies no penalty. Yellow Point: A query slightly off the manifold. This triggers the a