[Paper Review] On Reward-Free RL with Kernel and Neural Function Approximations: Single-Agent MDP and Markov Game

Shuang Qiu, Jieping Ye+2 more|arXiv (Cornell University)|Oct 19, 2021

Reinforcement Learning in Robotics4 citations

TL;DR

This paper proposes the first provably efficient reward-free reinforcement learning algorithms using kernel and neural function approximations for both single-agent MDPs and zero-sum Markov games. By employing an optimistic value iteration with exploration bonuses derived from function approximators, the method achieves $\widetilde{\mathcal{O}}(1/\varepsilon^{2})$ sample complexity to obtain an $\varepsilon$-suboptimal policy or $\varepsilon$-approximate Nash equilibrium for any extrinsic reward.

ABSTRACT

To achieve sample efficiency in reinforcement learning (RL), it necessitates efficiently exploring the underlying environment. Under the offline setting, addressing the exploration challenge lies in collecting an offline dataset with sufficient coverage. Motivated by such a challenge, we study the reward-free RL problem, where an agent aims to thoroughly explore the environment without any pre-specified reward function. Then, given any extrinsic reward, the agent computes the policy via a planning algorithm with offline data collected in the exploration phase. Moreover, we tackle this problem under the context of function approximation, leveraging powerful function approximators. Specifically, we propose to explore via an optimistic variant of the value-iteration algorithm incorporating kernel and neural function approximations, where we adopt the associated exploration bonus as the exploration reward. Moreover, we design exploration and planning algorithms for both single-agent MDPs and zero-sum Markov games and prove that our methods can achieve $\widetilde{\mathcal{O}}(1 /\varepsilon^2)$ sample complexity for generating a $\varepsilon$-suboptimal policy or $\varepsilon$-approximate Nash equilibrium when given an arbitrary extrinsic reward. To the best of our knowledge, we establish the first provably efficient reward-free RL algorithm with kernel and neural function approximators.

Motivation & Objective

• To address the challenge of sample-efficient exploration in offline reinforcement learning without pre-specified rewards.
• To develop a unified framework for reward-free RL that supports nonlinear function approximators like kernels and neural networks.
• To extend provably efficient algorithms from tabular and linear settings to nonlinear function approximation in both single-agent MDPs and Markov games.
• To design exploration and planning algorithms that achieve optimal sample complexity under arbitrary extrinsic rewards.

Proposed method

• Propose an optimistic variant of least-squares value iteration using kernel and neural function approximators to generate exploration bonuses as intrinsic rewards.
• Use the associated (scaled) bonus from the function approximation as the exploration reward during the offline data collection phase.
• Design a planning phase that computes policies or Nash equilibria using only offline data and any given extrinsic reward.
• For Markov games, the planning phase involves solving matrix games formed by Q-functions, which is computationally efficient.
• Leverage concentration inequalities and regularization in kernel and neural function spaces to bound estimation errors and ensure generalization.
• Integrate uncertainty quantification via regularization and confidence bounds to guide exploration in nonlinear function approximation settings.

Experimental results

Research questions

• RQ1 Can we design a provably efficient reward-free RL algorithm using kernel and neural function approximations for single-agent MDPs?
• RQ2 Can the proposed method achieve $\widetilde{\mathcal{O}}(1/\varepsilon^{2})$ sample complexity for learning $\varepsilon$-suboptimal policies in nonlinear function approximation settings?
• RQ3 Can the framework be extended to multi-agent zero-sum Markov games with similar sample efficiency guarantees?
• RQ4 How can exploration bonuses be effectively designed to align with nonlinear function approximators in the absence of extrinsic rewards?

Key findings

• The proposed algorithm achieves $\widetilde{\mathcal{O}}(1/\varepsilon^{2})$ sample complexity for generating an $\varepsilon$-suboptimal policy in single-agent MDPs with kernel and neural function approximations.
• For zero-sum Markov games, the method achieves $\widetilde{\mathcal{O}}(1/\varepsilon^{2})$ sample complexity to compute an $\varepsilon$-approximate Nash equilibrium.
• The planning phase in Markov games involves solving matrix games formed by Q-functions, which is computationally efficient and of independent interest.
• The theoretical analysis establishes that the algorithm's error bounds scale as $\mathcal{O}\left(\beta\sqrt{H^{4}[\Gamma(K,\lambda;\ker_{m})+\log(KH)]}/\sqrt{K}+H^{2}\beta\iota\right)$, with high probability.
• The method is the first to provide provable sample efficiency in reward-free RL with nonlinear function approximators, including kernels and neural networks.