[论文解读] Model Reduction Techniques for Computing Approximately Optimal Solutions for Markov Decision Processes

Thomas Dean, Robert Givan+1 more|arXiv (Cornell University)|Feb 6, 2013

Formal Methods in Verification参考文献 19被引用 104

一句话总结

本文引入了ε-同质状态空间划分，以将大型隐式马尔可夫决策过程（MDPs）简化为更小的有界参数MDP（BMDPs），从而近似原问题。通过借鉴形式验证中的模型约简技术，该方法能够高效计算具有可控误差边界的近似最优策略，实现解决方案质量与状态空间规模及计算成本之间的权衡。

ABSTRACT

We present a method for solving implicit (factored) Markov decision processes (MDPs) with very large state spaces. We introduce a property of state space partitions which we call epsilon-homogeneity. Intuitively, an epsilon-homogeneous partition groups together states that behave approximately the same under all or some subset of policies. Borrowing from recent work on model minimization in computer-aided software verification, we present an algorithm that takes a factored representation of an MDP and an 0<=epsilon<=1 and computes a factored epsilon-homogeneous partition of the state space. This partition defines a family of related MDPs - those MDPs with state space equal to the blocks of the partition, and transition probabilities "approximately" like those of any (original MDP) state in the source block. To formally study such families of MDPs, we introduce the new notion of a "bounded parameter MDP" (BMDP), which is a family of (traditional) MDPs defined by specifying upper and lower bounds on the transition probabilities and rewards. We describe algorithms that operate on BMDPs to find policies that are approximately optimal with respect to the original MDP. In combination, our method for reducing a large implicit MDP to a possibly much smaller BMDP using an epsilon-homogeneous partition, and our methods for selecting actions in BMDPs constitute a new approach for analyzing large implicit MDPs. Among its advantages, this new approach provides insight into existing algorithms to solving implicit MDPs, provides useful connections to work in automata theory and model minimization, and suggests methods, which involve varying epsilon, to trade time and space (specifically in terms of the size of the corresponding state space) for solution quality.

研究动机与目标

• 解决状态空间不可行的大规模隐式MDP的挑战。
• 开发一种在保持策略近似最优性的同时减小状态空间规模的方法。
• 为不确定环境下鲁棒策略计算的形式化有界参数MDP（BMDPs）概念。
• 通过可调ε参数，实现计算成本、内存使用与解决方案质量之间的权衡。
• 将MDP求解技术与模型最小化及自动机理论相连接，以提升可扩展性。

提出的方法

• 引入ε-同质状态划分的概念，即同一区块中的状态在一组策略下表现近似相同。
• 开发一种算法，从因子化MDP表示中计算因子化、ε-同质划分。
• 通过将状态聚合为区块构建BMDP，其中转移和奖励概率以原始MDP区块导出的区间进行有界处理。
• 应用BMDP求解算法，以找到对原始MDP近似最优的策略。
• 利用有界参数框架，确保从简化模型中得出的策略保持性能保证。
• 采用参数ε以控制近似精度与模型大小之间的权衡。

实验结果

研究问题

• RQ1 是否可以构建状态空间划分，使得每个区块内的状态在相关策略下表现近似一致？
• RQ2 如何在保持解决方案质量的前提下，将大型隐式MDP约简为更小的有界参数MDP？
• RQ3 与原始MDP相比，可在简化BMDP上计算的策略性能提供何种形式保证？
• RQ4 如何系统性地控制计算效率与解决方案准确度之间的权衡？
• RQ5 MDP模型约简与形式验证中模型最小化技术之间存在何种联系？

主要发现

• 该方法成功利用ε-同质划分将大型隐式MDP约简为显著更小的BMDPs。
• 在简化BMDP上计算出的策略被保证对原始MDP近似最优，且误差受ε控制。
• 该方法支持对原本因状态空间规模过大而无法求解的MDP实现可扩展求解。
• 通过ε参数，该框架支持在解决方案质量与计算成本之间进行系统性权衡。
• 与模型最小化的联系提供了理论基础，并为划分计算提供了实用算法。
• 该方法为现有MDP算法提供了新见解，并为可扩展强化学习指明了新方向。