[Paper Review] Lessons from AlphaZero for Optimal, Model Predictive, and Adaptive Control

Dimitri P. Bertsekas|arXiv (Cornell University)|Aug 20, 2021

Advanced Control Systems Optimization33 citations

TL;DR

The paper analyzes AlphaZero/TD-Gammon ideas of value-space approximation and rollouts and shows their broad applicability to deterministic and stochastic optimal control, including integration with MPC, adaptive, and decentralized control.

ABSTRACT

In this paper we aim to provide analysis and insights (often based on visualization), which explain the beneficial effects of on-line decision making on top of off-line training. In particular, through a unifying abstract mathematical framework, we show that the principal AlphaZero/TD-Gammon ideas of approximation in value space and rollout apply very broadly to deterministic and stochastic optimal control problems, involving both discrete and continuous search spaces. Moreover, these ideas can be effectively integrated with other important methodologies such as model predictive control, adaptive control, decentralized control, discrete and Bayesian optimization, neural network-based value and policy approximations, and heuristic algorithms for discrete optimization.

Motivation & Objective

• Motivate the study by connecting online decision making with offline training in control.
• Provide a unifying abstract framework showing how value-space approximation and rollouts transfer to control problems.
• Demonstrate integration of AlphaZero ideas with model predictive control, adaptive control, and decentralized control.
• Discuss connections to discrete and Bayesian optimization and neural network-based value and policy approximations.

Proposed method

• Propose a unifying abstract mathematical framework.
• Show that AlphaZero/TD-Gammon ideas of value-space approximation apply to both deterministic and stochastic optimal control.
• Demonstrate applicability to discrete and continuous search spaces.
• Illustrate integration with model predictive control, adaptive control, and decentralized control.
• Incorporate neural network-based value and policy approximations and heuristic optimization methods.

Experimental results

Research questions

• RQ1 How do AlphaZero/TD-Gammon ideas generalize to optimal control problems beyond game playing?
• RQ2 Can value-space approximation and rollout concepts be integrated with MPC, adaptive control, and decentralized control frameworks?
• RQ3 What is the role of discrete versus continuous search spaces in these generalized methods?
• RQ4 How do these ideas interact with neural network-based approximations and Bayesian optimization techniques?

Key findings

• Value-space approximation and rollout concepts from AlphaZero/TD-Gammon generalize to deterministic and stochastic optimal control.
• These ideas can be effectively integrated with MPC, adaptive control, decentralized control, and optimization methods.
• The framework applies to both discrete and continuous search spaces.
• Neural network-based value and policy approximations can be incorporated within the unified approach.
• The methodology connects with discrete and Bayesian optimization and heuristic discrete optimization techniques.