[Paper Review] Almost Optimal Algorithms for Two-player Markov Games with Linear Function Approximation.

Zixiang Chen, Dongruo Zhou+1 more|arXiv (Cornell University)|Feb 15, 2021

Reinforcement Learning in Robotics49 references4 citations

TL;DR

This paper proposes Nash-UCRL-VTR, an optimistic reinforcement learning algorithm for two-player zero-sum Markov games with linear function approximation. It achieves an $˜{O}(dH\sqrt{T})$ regret bound that matches a corresponding lower bound, proving near-optimality in regret for learning Nash equilibrium under linear transition models.

ABSTRACT

We study reinforcement learning for two-player zero-sum Markov games with simultaneous moves in the finite-horizon setting, where the transition kernel of the underlying Markov games can be parameterized by a linear function over the current state, both players' actions and the next state. In particular, we assume that we can control both players and aim to find the Nash Equilibrium by minimizing the duality gap. We propose an algorithm Nash-UCRL-VTR based on the principle Optimism-in-Face-of-Uncertainty. Our algorithm only needs to find a Coarse Correlated Equilibrium (CCE), which is computationally very efficient. Specifically, we show that Nash-UCRL-VTR can provably achieve an $ ilde{O}(dH\sqrt{T})$ regret, where $d$ is the linear function dimension, $H$ is the length of the game and $T$ is the total number of steps in the game. To access the optimality of our algorithm, we also prove an $ ilde{\Omega}( dH\sqrt{T})$ lower bound on the regret. Our upper bound matches the lower bound up to logarithmic factors, which suggests the optimality of our algorithm.

Motivation & Objective

• To develop a provably efficient algorithm for learning Nash equilibrium in two-player zero-sum Markov games with simultaneous moves.
• To handle environments where the transition kernel is linearly parameterized in state, actions, and next state.
• To minimize the duality gap by leveraging optimism in the face of uncertainty.
• To ensure computational efficiency by relying only on Coarse Correlated Equilibrium computation.
• To establish tight regret bounds that match information-theoretic lower bounds up to logarithmic factors.

Proposed method

• The algorithm employs the principle of Optimism-in-Face-of-Uncertainty (OFU) to balance exploration and exploitation.
• It uses linear function approximation to model the transition kernel as a linear function of state, both players' actions, and next state.
• The method computes a Coarse Correlated Equilibrium (CCE) at each stage, which is computationally efficient and sufficient for convergence to Nash equilibrium.
• It maintains confidence intervals over the value function estimates to guide exploration.
• The algorithm updates policies based on optimistic value estimates to minimize regret.
• The regret analysis relies on variance-aware concentration inequalities and a novel decomposition of the duality gap.

Experimental results

Research questions

• RQ1 Can we design a provably efficient algorithm for two-player Markov games with linear function approximation?
• RQ2 What is the optimal regret achievable in such games under linear transition models?
• RQ3 Can the algorithm achieve near-optimality in regret while maintaining computational tractability?
• RQ4 How does the use of Coarse Correlated Equilibrium compare to full Nash equilibrium computation in terms of efficiency and performance?
• RQ5 What is the information-theoretic lower bound on regret in this setting?

Key findings

• The proposed algorithm Nash-UCRL-VTR achieves an $˜{O}(dH\sqrt{T})$ regret bound, where $d$ is the linear function dimension, $H$ is the horizon, and $T$ is the total number of steps.
• The regret bound matches a corresponding $˜{\Omega}(dH\sqrt{T})$ lower bound, indicating near-optimality up to logarithmic factors.
• The algorithm only requires computing a Coarse Correlated Equilibrium at each step, ensuring computational efficiency.
• The use of optimism in the face of uncertainty enables effective exploration without requiring explicit Nash equilibrium computation.
• The theoretical analysis establishes tight regret bounds under linear function approximation for two-player zero-sum Markov games with simultaneous moves.
• The results demonstrate that the algorithm is optimal in terms of regret scaling, up to logarithmic factors.