[Paper Review] Model-based Path Integral Stochastic Control: A Bayesian Nonparametric Approach

Yunpeng Pan, Evangelos A. Theodorou+1 more|arXiv (Cornell University)|Dec 9, 2014

Gaussian Processes and Bayesian Inference15 references5 citations

TL;DR

This paper proposes a model-based, Bayesian nonparametric stochastic optimal control framework using Gaussian processes and path integral formulations to learn time-varying optimal controls from limited data. By leveraging analytic path integral solutions and iterative importance sampling, the method achieves superior data efficiency and learning speed compared to sampling-based path integral control and GP-based policy search, especially in high-dimensional and complex tasks like cart-pole and cart-double pendulum swing-up.

ABSTRACT

Over the last few years, sampling-based stochastic optimal control (SOC) frameworks have shown impressive performances in reinforcement learning (RL) with applications in robotics. However, such approaches require a large amount of samples from many interactions with the physical systems. To improve learning efficiency, we present a novel model-based and data-driven SOC framework based on path integral formulation and Gaussian processes (GPs). The proposed approach learns explicit and time-varying optimal controls autonomously from limited sampled data. Based on this framework, we propose an iterative control scheme with improved applicability in higher-dimensional and more complex control tasks. We demonstrate the effectiveness and efficiency of the proposed framework using two nontrivial examples. Compared to state-of-the-art RL methods, the proposed framework features superior control learning efficiency.

Motivation & Objective

• Address the data inefficiency of sampling-based stochastic optimal control (SOC) methods in high-dimensional robotic systems.
• Overcome the limitations of fixed policy parameterizations in existing SOC frameworks by enabling nonparametric, model-based learning of optimal controls.
• Improve learning speed and data efficiency by integrating Gaussian process dynamics models with analytic path integral solutions.
• Develop an iterative control scheme that enhances performance on complex, underactuated systems where uncontrolled dynamics sampling is insufficient.
• Demonstrate superior performance in terms of data consumption, computational time, and control quality compared to state-of-the-art methods like PILCO and iterative PI.

Proposed method

• Formulate the stochastic optimal control problem using a continuous-time SDE with unknown drift and diffusion matrices, modeled nonparametrically via Gaussian processes.
• Apply the Feynman-Kac formula to transform the Hamilton-Jacobi-Bellman equation into a path integral representation, enabling analytic computation of optimal controls.
• Use Gaussian processes to learn the dynamics model from limited sampled trajectories, incorporating model uncertainty in a Bayesian framework.
• Implement an iterative control scheme based on importance sampling, where controls are updated using samples from the controlled dynamics rather than the uncontrolled ones.
• Derive explicit, time-varying optimal control laws through analytic evaluation of path integrals, avoiding numerical PDE solves or iterative optimization.
• Introduce two variants: GPPI (based on uncontrolled dynamics sampling) and iGPPI (iterative refinement using controlled dynamics), both leveraging analytic path integral solutions.

Experimental results

Research questions

• RQ1 Can a Bayesian nonparametric model-based approach improve data efficiency in stochastic optimal control compared to sampling-based path integral methods?
• RQ2 How does incorporating Gaussian process dynamics models with analytic path integral solutions affect learning speed and control performance in high-dimensional systems?
• RQ3 To what extent does iterative refinement using importance sampling from controlled dynamics enhance performance on complex, underactuated tasks?
• RQ4 How does the proposed framework compare in data efficiency and computational cost to PILCO and iterative PI in challenging control tasks?
• RQ5 Can the framework learn optimal controls without prior policy parameterization or reliance on external optimization solvers?

Key findings

• GPPI and iGPPI achieve comparable optimal control performance to iterative PI control while requiring significantly fewer sampled data points and less total computational time.
• iGPPI outperforms GPPI in complex tasks like the cart-double inverted pendulum swing-up, where sampling from uncontrolled dynamics is insufficient for convergence.
• The proposed framework reduces data consumption and learning time compared to sampling-based PI control, demonstrating superior data efficiency.
• Compared to PILCO, the proposed method achieves faster learning speed due to the absence of iterative policy optimization, despite PILCO’s superior data efficiency in some cases.
• The iterative scheme (iGPPI) enables better terminal cost reduction in high-dimensional, underactuated systems, confirming its enhanced applicability to complex control tasks.
• The analytic path integral formulation enables explicit, closed-form control laws without numerical PDE solves or gradient-based optimization, enhancing computational efficiency.