[论文解读] ARES: Adaptive Receding-Horizon Synthesis of Optimal Plans

Anna Lukina, Lukas Esterle+6 more|arXiv (Cornell University)|Dec 21, 2016

Reinforcement Learning in Robotics参考文献 27被引用 6

一句话总结

ARES 是一种自适应的后退时域算法，通过动态调整时域长度和粒子群大小的粒子群优化（PSO）方法，为马尔可夫决策过程（MDP）合成最优计划。它通过借鉴重要性分割的分层路标点实现收敛保证，在平均63秒内成功生成7只鸟组成的V字形编队计划，成功率高达95%。

ABSTRACT

We introduce ARES, an efficient approximation algorithm for generating optimal plans (action sequences) that take an initial state of a Markov Decision Process (MDP) to a state whose cost is below a specified (convergence) threshold. ARES uses Particle Swarm Optimization, with adaptive sizing for both the receding horizon and the particle swarm. Inspired by Importance Splitting, the length of the horizon and the number of particles are chosen such that at least one particle reaches a next-level state, that is, a state where the cost decreases by a required delta from the previous-level state. The level relation on states and the plans constructed by ARES implicitly define a Lyapunov function and an optimal policy, respectively, both of which could be explicitly generated by applying ARES to all states of the MDP, up to some topological equivalence relation. We also assess the effectiveness of ARES by statistically evaluating its rate of success in generating optimal plans. The ARES algorithm resulted from our desire to clarify if flying in V-formation is a flocking policy that optimizes energy conservation, clear view, and velocity alignment. That is, we were interested to see if one could find optimal plans that bring a flock from an arbitrary initial state to a state exhibiting a single connected V-formation. For flocks with 7 birds, ARES is able to generate a plan that leads to a V-formation in 95% of the 8,000 random initial configurations within 63 seconds, on average. ARES can also be easily customized into a model-predictive controller (MPC) with an adaptive receding horizon and statistical guarantees of convergence. To the best of our knowledge, our adaptive-sizing approach is the first to provide convergence guarantees in receding-horizon techniques.

研究动机与目标

• 开发一种可扩展、自适应的算法，用于在MDP中合成最优动作序列，并收敛至目标成本阈值。
• 解决固定时域模型预测控制（MPC）的局限性，包括缺乏收敛保证和窗口尺寸不优的问题。
• 将重要性分割与PSO结合，通过分层成本等级引导优化，确保鲁棒收敛。
• 为复杂动力系统（如鸟群或无人机编队）实现实时、在线策略合成。
• 通过基于分层的状态推进，提供统计收敛保证和隐式李雅普诺夫函数构造。

提出的方法

• ARES采用基于分层的后退时域框架，将状态按递减至目标阈值的成本等级进行划分。
• 根据到达下一成本等级的难度，自适应地增加PSO的规划时域和粒子数量。
• 每一等级对应一个状态，其成本相比前一等级减少预设的delta值，确保向目标的持续进展。
• 在每一等级应用粒子群优化以生成候选动作序列，适应度由成本降低程度决定。
• 通过等级结构隐式定义李雅普诺夫函数，并通过构建的计划隐式定义最优策略。
• 通过动态调整时域和粒子群大小，ARES可转化为具有统计收敛保证的模型预测控制器。

实验结果

研究问题

• RQ1 在缺乏最优时域长度先验知识的情况下，基于PSO的自适应后退时域方法能否在MDP规划中保证收敛？
• RQ2 如何自适应地调节规划时域和粒子数量，以在复杂MDP中平衡效率与成功率？
• RQ3 源自重要性分割的分层路标点能否提升PSO在避免局部极小值方面的性能，从而优化计划合成？
• RQ4 ARES在多大程度上能够以统计可靠性生成真实系统（如鸟群或无人机编队）的最优计划？
• RQ5 能否从ARES的基于分层的结构中显式恢复隐式李雅普诺夫函数和最优策略？

主要发现

• 对于7只鸟的鸟群，ARES在8,000个随机初始配置中，95%的情况下成功生成了最优V字形编队计划。
• 每条计划的平均执行时间仅为63秒，展现出强大的计算效率。
• ARES通过基于成本等级推进的自适应时域和粒子群大小，实现了收敛保证。
• 基于分层的结构隐式定义了李雅普诺夫函数，确保系统稳定并收敛至低代价状态。
• 该方法通过动态调整规划深度和粒子数量，避免局部极小值，优于固定时域MPC。
• ARES可扩展为具有统计收敛保证的实时模型预测控制器。