[论文解读] Accelerated Online Risk-Averse Policy Evaluation in POMDPs with Theoretical Guarantees and Novel CVaR Bounds

Yaacov Pariente, Vadim Indelman|arXiv (Cornell University)|Feb 26, 2026

Reinforcement Learning in Robotics被引用 0

一句话总结

该论文推导CVaR界限以在简化的信念MDP下界定风险厌恶的价值函数，开发具有粒子基础框架保证的在线估计量，并利用这些界限通过动作消除安全地加速规划。

ABSTRACT

Risk-averse decision-making under uncertainty in partially observable domains is a central challenge in artificial intelligence and is essential for developing reliable autonomous agents. The formal framework for such problems is the partially observable Markov decision process (POMDP), where risk sensitivity is introduced through a risk measure applied to the value function, with Conditional Value-at-Risk (CVaR) being a particularly significant criterion. However, solving POMDPs is computationally intractable in general, and approximate methods rely on computationally expensive simulations of future agent trajectories. This work introduces a theoretical framework for accelerating CVaR value function evaluation in POMDPs with formal performance guarantees. We derive new bounds on the CVaR of a random variable X using an auxiliary random variable Y, under assumptions relating their cumulative distribution and density functions; these bounds yield interpretable concentration inequalities and converge as the distributional discrepancy vanishes. Building on this, we establish upper and lower bounds on the CVaR value function computable from a simplified belief-MDP, accommodating general simplifications of the transition dynamics. We develop estimators for these bounds within a particle-belief MDP framework with probabilistic guarantees, and employ them for acceleration via action elimination: actions whose bounds indicate suboptimality under the simplified model are safely discarded while ensuring consistency with the original POMDP. Empirical evaluation across multiple POMDP domains confirms that the bounds reliably separate safe from dangerous policies while achieving substantial computational speedups under the simplified model.

研究动机与目标

• 使用辅助变量Y在分布差异下推导随机变量X的CVaR界限。
• 将原始CVaR价值函数与简化信念-MMDP价值函数联系起来并给出可证明的界限。
• 在粒子信念MDP中开发在线估计量以计算这些界限，并给出概率保证。
• 将界限应用于通过安全的动作消除来加速规划，同时保持性能。

提出的方法

• 推导将X与Y相关的统一和非统一CVaR界限（定理5.1–5.4）。
• 表征原始与简化信念模型之间的epsilon-差异界限。
• 将风险厌恶的POMDPs表述为以CVaR为目标（V_M(b_k, α) 和 Q_M(b_k,a_k,α)）。
• 在粒子信念MDP（PB-MDP）中开发用于界限的在线估计量并证明概率性能保证（定理7.4）。
• 利用界限在在线规划过程中对次优动作进行剪枝并展示加速效果。
• 给出CVaR估计值的收敛界限（定理3.1及相关结果）。

(a)

实验结果

研究问题

• RQ1 如何利用可处理的简化模型对POMDP中的回报CVaR进行有界？
• RQ2 在原始与简化动态之间的分布差距条件下，哪些条件可确保CVaR界限有信息量？
• RQ3 在粒子信念框架中在线估计量能否对这些CVaR界限提供概率保证？
• RQ4 基于CVaR界限的动作消除策略是否能在几乎不损失性能的前提下带来计算加速？

主要发现

• 确立了统一的CVaR界限：X和Y通过ε差异进行界定，且对α有条件（定理5.1）。
• 证明了当ε→0时界限收敛（定理5.2）。
• 引入使用函数g(x)的更紧下界构造（定理5.3）和基于密度差的界限（定理5.4）。
• 推导了CVaR估计的收敛性界限，使基于样本的保证成为可能（定理5.5及相关）。
• 在多个POMDP领域通过动作消除实现了显著的计算加速，且策略退化很小。

(b)