[论文解读] Controlled Online Optimization Learning (COOL): Finding the ground state of spin Hamiltonians with reinforcement learning

Kyle Mills, Pooya Ronagh+1 more|arXiv (Cornell University)|Feb 28, 2020

Neural Networks and Reservoir Computing参考文献 52被引用 5

一句话总结

本文提出了受控在线优化学习（COOL），一种强化学习（RL）框架，可自主学习自旋哈密顿量中模拟退火的最优温度调度。通过近端策略优化，RL智能体在性能上超越了标准启发式调度，可在不同哈密顿量类别间实现泛化，并在L=14²系统上实现接近两个数量级的性能提升，通过动态调节温度以逃逸局部极小值并达到基态。

ABSTRACT

Reinforcement learning (RL) has become a proven method for optimizing a procedure for which success has been defined, but the specific actions needed to achieve it have not. We apply the so-called "black box" method of RL to what has been referred as the "black art" of simulated annealing (SA), demonstrating that an RL agent based on proximal policy optimization can, through experience alone, arrive at a temperature schedule that surpasses the performance of standard heuristic temperature schedules for two classes of Hamiltonians. When the system is initialized at a cool temperature, the RL agent learns to heat the system to "melt" it, and then slowly cool it in an effort to anneal to the ground state; if the system is initialized at a high temperature, the algorithm immediately cools the system. We investigate the performance of our RL-driven SA agent in generalizing to all Hamiltonians of a specific class; when trained on random Hamiltonians of nearest-neighbour spin glasses, the RL agent is able to control the SA process for other Hamiltonians, reaching the ground state with a higher probability than a simple linear annealing schedule. Furthermore, the scaling performance (with respect to system size) of the RL approach is far more favourable, achieving a performance improvement of one order of magnitude on L=14x14 systems. We demonstrate the robustness of the RL approach when the system operates in a "destructive observation" mode, an allusion to a quantum system where measurements destroy the state of the system. The success of the RL agent could have far-reaching impact, from classical optimization, to quantum annealing, to the simulation of physical systems.

研究动机与目标

• 开发一种强化学习方法，自主学习自旋玻璃系统中模拟退火的最优温度调度。
• 实现所学策略在同类别不同哈密顿量间的泛化，而无需事先了解哈密顿量结构。
• 与经典线性调度相比，改善模拟退火在系统规模增大时的缩放性能。
• 评估在破坏性观测条件下（与量子及物理实验相关）RL方法的鲁棒性。
• 证明RL可为统计物理中的复杂优化任务发现非平凡、自适应的控制策略。

提出的方法

• 该方法采用近端策略优化（PPO）智能体，在模拟退火过程中控制温度调度。
• 智能体在每一步观测系统的能量和自旋构型，将该过程视为马尔可夫决策过程，目标是最小化能量。
• 卷积神经网络处理自旋构型以估计价值函数和策略，从而实现与系统规模的可扩展性。
• 智能体通过试错学习，根据最终状态是否为基态接收稀疏奖励。
• 训练采用“破坏性观测”模式，模拟测量导致系统态坍缩的情景。
• 策略在固定和随机生成的哈密顿量上进行评估，包括近邻自旋玻璃和弱-强簇模型。

实验结果

研究问题

• RQ1 强化学习智能体是否能在不了解哈密顿量结构的前提下，学习到有效的模拟退火温度调度？
• RQ2 RL智能体在同类别不同自旋哈密顿量实例间的策略泛化能力如何？
• RQ3 随着系统规模增大，基于RL的退火方法与经典线性调度相比，其缩放行为如何？
• RQ4 在测量导致系统态坍缩的破坏性观测条件下，RL智能体表现如何？
• RQ5 RL智能体能否发现非平凡策略，例如在冷却前先加热以逃逸局部极小值？

主要发现

• 在多个哈密顿量类别中，RL智能体在达到基态方面始终优于标准启发式温度调度。
• 在L=14²系统上，RL方法相较于线性退火调度实现了接近两个数量级的性能提升。
• 智能体学会在低温初始化时对系统加热以逃逸局部极小值，随后逐步冷却以达到基态。
• 该策略能有效泛化到同一类别中未见过的新哈密顿量，即使耦合项为随机生成且训练时未出现。
• 智能体所学习的价值函数直观地表征了相空间，将更高价值赋予靠近基态的构型。
• 该方法在破坏性观测条件下仍保持鲁棒性，证明其在测量成本高或具有破坏性的物理与量子应用中的可行性。