[Paper Review] On Reward-Free Reinforcement Learning with Linear Function Approximation

Ruosong Wang, Simon S. Du+2 more|arXiv (Cornell University)|Jun 19, 2020

Reinforcement Learning in RoboticsComputer Science36 references33 citations

TL;DR

The paper analyzes reward-free RL with linear function approximation, proving a polynomial upper bound under linear MDPs and an exponential lower bound under linear Q*, plus an explicit hardness separation between model-based and value-based assumptions.

ABSTRACT

Reward-free reinforcement learning (RL) is a framework which is suitable for both the batch RL setting and the setting where there are many reward functions of interest. During the exploration phase, an agent collects samples without using a pre-specified reward function. After the exploration phase, a reward function is given, and the agent uses samples collected during the exploration phase to compute a near-optimal policy. Jin et al. [2020] showed that in the tabular setting, the agent only needs to collect polynomial number of samples (in terms of the number states, the number of actions, and the planning horizon) for reward-free RL. However, in practice, the number of states and actions can be large, and thus function approximation schemes are required for generalization. In this work, we give both positive and negative results for reward-free RL with linear function approximation. We give an algorithm for reward-free RL in the linear Markov decision process setting where both the transition and the reward admit linear representations. The sample complexity of our algorithm is polynomial in the feature dimension and the planning horizon, and is completely independent of the number of states and actions. We further give an exponential lower bound for reward-free RL in the setting where only the optimal $Q$-function admits a linear representation. Our results imply several interesting exponential separations on the sample complexity of reward-free RL.

Motivation & Objective

• Investigate whether reward-free RL with linear function approximation can achieve provable efficiency.
• Characterize sample complexity under two modeling assumptions: linear MDP and linear Q*.
• Explore hardness results to understand limits of reward-free RL with function approximation.
• Provide insights into separations between model-based and value-based settings in reward-free RL.

Proposed method

• Propose an algorithm for reward-free RL in linear MDPs that collects 1/B(d^3 H^6 / 8 B^2) trajectories during exploration.
• Use an exploration-driven upper-confidence-bound (UCB) bonus to construct an exploration reward during the exploration phase.
• Leverage least-squares value iteration (LSVI) to estimate Q-functions and derive optimistic value functions.
• Planning phase uses the dataset to perform batch RL with a uniform covering of the observed transitions and an optimistic Q-function.
• Prove optimistic planning guarantees via concentration arguments and the elliptical potential lemma.
• Demonstrate a lower bound for reward-free RL under the linear Q* assumption, showing exponential sample complexity in exploration.

Experimental results

Research questions

• RQ1 Can reward-free RL be efficiently solved with linear function approximation under the linear MDP assumption?
• RQ2 What is the sample complexity of reward-free RL when only the optimal Q* is linear (linear Q*), and how does it compare to the linear MDP setting?
• RQ3 Do hardness results exist under weaker assumptions, and do exponential separations arise between model-based and value-based settings?
• RQ4 How does the presence of a simulator (generative model) affect sample complexity compared to the standard RL model in reward-free RL?
• RQ5 What conceptual separations emerge between reward-free RL and standard RL under linear function approximation?

Key findings

• Under linear MDP, reward-free RL achieves a polynomial exploration complexity of rom source: O(d^3 H^6 / ^2) trajectories, yielding -optimal policies for arbitrary reward functions in planning with high probability.
• An exploration-driven reward function r_h^k is constructed as u_h^k / H to incentivize visiting uncertain state-action pairs.
• If only Q* is linear (linear Q*), any reward-free RL algorithm requires exponential samples in exploration to guarantee near-optimal planning, even in deterministic MDPs.
• There exists an exponential separation between reward-free RL and standard RL under the linear Q* assumption, as standard RL can achieve polynomial planning with sufficient conditions.
• With a simulator (generative model), there is a polynomial upper bound under linear Q*, indicating an exponential separation between simulator and non-simulator settings.
• The results establish that reward-free RL can be exponentially harder than standard RL in the same function-approximation regime and that simulators can dramatically reduce sample complexity.