[Paper Review] Model-based Reinforcement Learning for Continuous Control with Posterior Sampling

Ying Fan, Yifei Ming|arXiv (Cornell University)|Nov 20, 2020

Advanced Bandit Algorithms Research4 citations

TL;DR

This paper proposes MPC-PSRL, a model-based reinforcement learning algorithm for continuous control that uses posterior sampling with Bayesian linear regression on neural network features to improve exploration. It achieves a regret bound of $\tilde{O}(H^{3/2}d\sqrt{T})$ in linear MDPs and outperforms prior model-based methods in sample efficiency while matching model-free performance at convergence.

ABSTRACT

Balancing exploration and exploitation is crucial in reinforcement learning (RL). In this paper, we study model-based posterior sampling for reinforcement learning (PSRL) in continuous state-action spaces theoretically and empirically. First, we show the first regret bound of PSRL in continuous spaces which is polynomial in the episode length to the best of our knowledge. With the assumption that reward and transition functions can be modeled by Bayesian linear regression, we develop a regret bound of $ ilde{O}(H^{3/2}d\sqrt{T})$, where $H$ is the episode length, $d$ is the dimension of the state-action space, and $T$ indicates the total time steps. This result matches the best-known regret bound of non-PSRL methods in linear MDPs. Our bound can be extended to nonlinear cases as well with feature embedding: using linear kernels on the feature representation $ϕ$, the regret bound becomes $ ilde{O}(H^{3/2}d_ϕ\sqrt{T})$, where $d_ϕ$ is the dimension of the representation space. Moreover, we present MPC-PSRL, a model-based posterior sampling algorithm with model predictive control for action selection. To capture the uncertainty in models, we use Bayesian linear regression on the penultimate layer (the feature representation layer $ϕ$) of neural networks. Empirical results show that our algorithm achieves the state-of-the-art sample efficiency in benchmark continuous control tasks compared to prior model-based algorithms, and matches the asymptotic performance of model-free algorithms.

Motivation & Objective

• To develop a theoretically grounded, computationally tractable method for exploration in continuous control via posterior sampling.
• To establish a regret bound for PSRL in continuous state-action spaces that is polynomial in episode length H and sub-linear in T.
• To improve sample efficiency in model-based RL by better capturing model uncertainty through Bayesian function approximation.
• To demonstrate that posterior sampling with model predictive control (MPC) can match or exceed model-free performance in continuous control tasks.

Proposed method

• Uses posterior sampling for reinforcement learning (PSRL) in continuous MDPs by sampling a single MDP from the posterior distribution per episode.
• Applies Bayesian linear regression to the penultimate layer of a neural network to model transition and reward functions, capturing uncertainty in the feature space.
• Employs model predictive control (MPC) for action selection, optimizing over the sampled MDP for the entire episode.
• Uses kernelized features (e.g., linear kernels on representation φ) to extend results to nonlinear MDPs with regret bound $\tilde{O}(H^{3/2}d_{\phi}\sqrt{T})$.
• Leverages the posterior distribution over models to naturally balance exploration and exploitation without optimizing over confidence sets.
• Employs a Bayesian approach to uncertainty quantification, avoiding the computational burden of UCB-style confidence set optimization.

Experimental results

Research questions

• RQ1 Can PSRL in continuous control achieve a regret bound that is polynomial in H and sub-linear in T, matching the best-known UCB-based bounds?
• RQ2 How does posterior sampling with Bayesian function approximation compare to UCB-based methods in terms of computational efficiency and statistical performance?
• RQ3 Can model-based RL with posterior sampling and MPC outperform existing model-based and model-free algorithms in sample efficiency on continuous control benchmarks?
• RQ4 What is the impact of modeling uncertainty via Bayesian linear regression on the feature representation layer in deep networks?

Key findings

• The paper establishes the first regret bound for PSRL in continuous MDPs of $\tilde{O}(H^{3/2}d\sqrt{T})$, matching the best-known UCB-based bounds in linear MDPs.
• The regret bound extends to nonlinear MDPs via feature embedding, yielding $\tilde{O}(H^{3/2}d_{\phi}\sqrt{T})$ when using linear kernels on the representation space.
• MPC-PSRL achieves state-of-the-art sample efficiency on stochastic continuous control benchmarks, including continuous Cartpole and Pendulum Swing Up with added noise.
• In 7-DOF Reacher and Pusher tasks, MPC-PSRL outperforms PETS and MBPO when oracle rewards are available, converging faster and achieving better final performance.
• When both dynamics and rewards are unknown, MPC-PSRL significantly outperforms prior model-based and model-free methods and matches the asymptotic performance of SAC.
• The results confirm that modeling uncertainty via posterior sampling on neural network features leads to better exploration than ensemble-based methods like PETS and MBPO.