[Paper Review] A Game Theoretic Framework for Model Based Reinforcement Learning

Aravind Rajeswaran, Igor Mordatch+1 more|arXiv (Cornell University)|Apr 16, 2020

Reinforcement Learning in Robotics67 references43 citations

TL;DR

The paper casts model-based reinforcement learning as a two-player game between a policy player and a model player, and solves it with Stackelberg-game-based algorithms (PAL and MAL) that achieve strong sample efficiency and scale to high-dimensional tasks.

ABSTRACT

Model-based reinforcement learning (MBRL) has recently gained immense interest due to its potential for sample efficiency and ability to incorporate off-policy data. However, designing stable and efficient MBRL algorithms using rich function approximators have remained challenging. To help expose the practical challenges in MBRL and simplify algorithm design from the lens of abstraction, we develop a new framework that casts MBRL as a game between: (1) a policy player, which attempts to maximize rewards under the learned model; (2) a model player, which attempts to fit the real-world data collected by the policy player. For algorithm development, we construct a Stackelberg game between the two players, and show that it can be solved with approximate bi-level optimization. This gives rise to two natural families of algorithms for MBRL based on which player is chosen as the leader in the Stackelberg game. Together, they encapsulate, unify, and generalize many previous MBRL algorithms. Furthermore, our framework is consistent with and provides a clear basis for heuristics known to be important in practice from prior works. Finally, through experiments we validate that our proposed algorithms are highly sample efficient, match the asymptotic performance of model-free policy gradient, and scale gracefully to high-dimensional tasks like dexterous hand manipulation. Additional details and code can be obtained from the project page at https://sites.google.com/view/mbrl-game

Motivation & Objective

• Expose practical challenges in model-based RL and unify algorithm design through abstraction.
• Cast MBRL as a two-player game between policy optimization and world-model fitting.
• Develop Stackelberg-based algorithms to compute equilibrium in continuous games.
• Demonstrate improved sample efficiency and scalability to high-dimensional tasks.
• Provide insights that connect and generalize prior MBRL approaches.

Proposed method

• Model-based RL is formulated as a two-player game: a policy player maximizes rewards in the learned model, and a model player minimizes prediction error under the policy’s induced state distribution.
• Adopt Stackelberg game structure to enable stable bi-level optimization and derive practical gradient-based updates.
• Introduce two leader-follower variants: Policy as Leader (PAL) and Model as Leader (MAL), each with a specific nested optimization scheme.
• Use first-order approximations to solve the bi-level updates, enabling iterative updates of either the model then policy (PAL) or policy then model (MAL).
• Represent the policy and the dynamics model with neural networks and employ ensembles and entropy regularization for robustness.

Experimental results

Research questions

• RQ1 Can viewing MBRL as a two-player game yield stable and efficient learning dynamics?
• RQ2 Do Stackelberg-based PAL and MAL algorithms improve sample efficiency and scalability over prior MBRL and model-free methods?
• RQ3 How do PAL and MAL compare in environments with changing dynamics or goal distributions?
• RQ4 What theoretical guarantees link equilibrium quality to policy optimality in the environment?

Key findings

• PAL and MAL exhibit stable, near-monotonic learning across continuous control tasks.
• Both PAL and MAL outperform prior model-based and model-free methods in sample efficiency and scale to high-dimensional dexterous manipulation tasks.
• PAL tends to learn faster than MAL in the studied tasks, while MAL better handles changes in goal distributions.
• BR (Best Response) leads to instability, while GDA (Gradient Descent-Ascent) is slow or unstable in some settings.
• The methods achieve asymptotic performance on par with model-free policy gradient baselines in the evaluated tasks.
• The framework unifies and generalizes prior MBRL approaches by linking conservatism and data aggregation through a principled game-theoretic lens.