[Paper Review] Scalable trust-region method for deep reinforcement learning using Kronecker-factored approximation

Yuhuai Wu, Elman Mansimov+3 more|arXiv (Cornell University)|Aug 17, 2017

Reinforcement Learning in RoboticsComputer Science22 references470 citations

TL;DR

ACKTR applies Kronecker-factored approximate curvature with trust-region natural gradient to actor-critic methods, achieving 2-3x improvements in sample efficiency on Atari and MuJoCo and enabling learning from raw pixel inputs.

ABSTRACT

In this work, we propose to apply trust region optimization to deep reinforcement learning using a recently proposed Kronecker-factored approximation to the curvature. We extend the framework of natural policy gradient and propose to optimize both the actor and the critic using Kronecker-factored approximate curvature (K-FAC) with trust region; hence we call our method Actor Critic using Kronecker-Factored Trust Region (ACKTR). To the best of our knowledge, this is the first scalable trust region natural gradient method for actor-critic methods. It is also a method that learns non-trivial tasks in continuous control as well as discrete control policies directly from raw pixel inputs. We tested our approach across discrete domains in Atari games as well as continuous domains in the MuJoCo environment. With the proposed methods, we are able to achieve higher rewards and a 2- to 3-fold improvement in sample efficiency on average, compared to previous state-of-the-art on-policy actor-critic methods. Code is available at https://github.com/openai/baselines

Motivation & Objective

• Motivate improved sample efficiency in deep reinforcement learning beyond standard SGD-based updates.
• Develop a scalable natural-gradient method suitable for large actor-critic models.
• Extend Kronecker-factored curvature to jointly optimize both actor and critic.
• Enable learning directly from raw pixel inputs in both discrete and continuous control tasks.

Proposed method

• Use Kronecker-factored approximate curvature (K-FAC) to invert the Fisher matrix efficiently for natural gradient updates.
• Apply the natural gradient with trust-region constraints to both the actor and the critic (Gauss-Newton for the critic).
• Construct a joint, optionally shared, architecture for actor-critic and sample outputs independently when needed.
• Incorporate factorized Tikhonov damping and asynchronous statistics/inverses to reduce computation.
• Tune step size using a trust-region formulation to limit KL divergence in updates.

Experimental results

Research questions

• RQ1 How does ACKTR compare with state-of-the-art on-policy methods and second-order baselines in terms of sample efficiency and computational efficiency?
• RQ2 What is the impact of applying natural-gradient updates to both actor and critic on stability and performance?
• RQ3 How does ACKTR scale with batch size and input modality (including pixel inputs) in discrete and continuous control?
• RQ4 Which norm and damping strategy for critic optimization best stabilizes training and improves sample efficiency?

Key findings

• ACKTR substantially improves sample efficiency and final performance over A2C and TRPO on Atari and MuJoCo benchmarks.
• Optimizing both actor and critic with natural-gradient updates yields scalable performance gains not achievable by previous methods.
• Using Gauss-Newton-based norm for the critic provides notable gains in sample efficiency and training stability compared with Euclidean-norm updates.
• ACKTR maintains computational cost close to SGD-based methods, with per-update cost only modestly higher.
• ACKTR demonstrates strong performance when learning from pixel inputs in continuous control tasks, including competing results from raw-pixel observations.
• Larger batch sizes benefit ACKTR more than first-order methods, indicating potential for significant speedups in distributed settings.