[论文解读] Optimal Matrix Momentum Stochastic Approximation and Applications to Q-learning

Adithya M. Devraj, Ana Bušić+1 more|arXiv (Cornell University)|Sep 17, 2018

Stochastic Gradient Optimization Techniques参考文献 25被引用 5

一句话总结

本文提出了PolSA和NeSA两种新颖的随机逼近算法，其在根查找问题中具有最优的渐近方差。通过引入受Polyak的动量法和Nesterov加速启发的矩阵动量，PolSA实现了与计算成本高昂的随机牛顿-拉夫森（Stochastic Newton-Raphson, SNR）方法相当的性能；而NeSA则提供了一种简化变体，在保证强收敛性的同时显著降低了计算成本，尤其在Q-learning和TD-learning应用中表现出色。

ABSTRACT

Acceleration is an increasingly common theme in the stochastic optimization literature. The two most common examples are Nesterov's method, and Polyak's momentum technique. In this paper two new algorithms are introduced for root finding problems: 1) PolSA is a root finding algorithm with specially designed matrix momentum, and 2) NeSA can be regarded as a variant of Nesterov's algorithm, or a simplification of PolSA. The PolSA algorithm is new even in the context of optimization (when cast as a root finding problem). The research surveyed in this paper is motivated by applications to reinforcement learning. It is well known that most variants of TD- and Q-learning may be cast as SA (stochastic approximation) algorithms, and the tools from general SA theory can be used to investigate convergence and bounds on convergence rate. In particular, the asymptotic variance is a common metric of performance for SA algorithms, and is also one among many metrics used in assessing the performance of stochastic optimization algorithms. There are two well known SA techniques that are known to have optimal asymptotic variance: the Ruppert-Polyak averaging technique, and stochastic Newton-Raphson (SNR). The former algorithm can have extremely bad transient performance, and the latter can be computationally expensive. It is demonstrated here that parameter estimates from the new PolSA algorithm couple with those of the ideal (but more complex) SNR algorithm. The new algorithm is thus a third approach to obtain optimal asymptotic covariance. These strong results require assumptions on the model. A linearized model is considered, and the noise is assumed to be a martingale difference sequence. Numerical results are obtained in a non-linear setting that is the motivation for this work: In PolSA implementations of Q-learning it is observed that coupling occurs with SNR in this non-ideal setting.

研究动机与目标

• 开发一种具有最优渐近方差的新随机逼近算法，避免随机牛顿-拉夫森（SNR）方法的计算开销。
• 设计一种基于矩阵动量的算法（PolSA），在线性和非线性设置下均能与SNR实现耦合，确保最优收敛速率。
• 提出一种简化变体（NeSA），在降低计算复杂度的同时保持强收敛性能。
• 将新算法应用于强化学习问题，特别是Q-learning和TD(0)-learning，其中标准随机逼近方法常因高方差或收敛缓慢而表现不佳。
• 在标准随机逼近假设下，建立收敛速率和渐近协方差的理论保证，包括鞅差噪声和线性化动力学。

提出的方法

• PolSA在随机逼近框架中通过时变矩阵序列{Gn}引入矩阵动量，其更新包含一个动量项∆θn和一个增益调整的梯度项。
• 该算法使用矩阵增益序列{Gn}，以近似逆Hessian结构，从而在无需显式计算Hessian逆矩阵的情况下模拟SNR的行为。
• NeSA是PolSA的简化变体，将矩阵增益替换为固定标量动量参数ζ，从而实现具有Nesterov式加速项的递归更新。
• 算法在一种线性化设置下进行分析，其中函数f(θ, X)被近似为Aθ − b，噪声被建模为鞅差序列。
• 理论分析表明，PolSA实现了最优渐近协方差Σ∗ = A−1Σ∆A−T，与SNR方法一致。
• 非线性Q-learning设置下的数值实验表明，PolSA与NeSA在显著降低计算成本的同时，性能接近SNR。

实验结果

研究问题

• RQ1 能否设计一种基于矩阵动量的随机逼近算法，在无需每步进行Hessian逆或矩阵逆运算的情况下，实现与随机牛顿-拉夫森（SNR）方法相同的最优渐近方差？
• RQ2 在非线性设置（如Q-learning）中，所提出的PolSA算法如何与SNR实现耦合，其中底层动力学具有非线性和马氏特性？
• RQ3 在强化学习的随机逼近中使用矩阵动量时，计算复杂度与收敛速率之间的性能权衡如何？
• RQ4 PolSA的简化变体（NeSA）是否能在计算效率更高、实现更简便的同时，保持SNR的收敛特性？
• RQ5 与标准随机逼近及经典Q-learning变体相比，PolSA与NeSA中的矩阵动量结构在TD(0)和Q-learning中对瞬态与渐近性能的提升程度如何？

主要发现

• PolSA实现了最优渐近协方差Σ∗ = A−1Σ∆A−T，达到渐近方差的理论下界，与随机牛顿-拉夫森（SNR）方法一致。
• 在非线性Q-learning设置中，PolSA与SNR表现出强耦合，表明其在问题非线性的情况下仍能实现近似最优的收敛速率。
• NeSA作为PolSA的简化变体，在Q-learning中表现出色，其收敛速度和方差降低性能优于标准方法及近期变体。
• PolSA与NeSA中的矩阵动量相比标准随机逼近和经典Q-learning算法，能实现更快的收敛速度和更低的渐近方差。
• 理论分析证实，PolSA与NeSA在标准假设下均能实现收敛，包括鞅差噪声和线性化动力学。
• 数值结果表明，即使在模型偏离线性化假设的情况下，PolSA与NeSA在实际强化学习任务中仍能保持优异性能。