[Paper Review] Simultaneous Perturbation Algorithms for Batch Off-Policy Search

Raphaël Fonteneau, L. A. Prashanth|arXiv (Cornell University)|Mar 18, 2014

Reinforcement Learning in Robotics14 references3 citations

TL;DR

This paper proposes novel batch off-policy reinforcement learning algorithms for continuous state and action spaces using simultaneous perturbation stochastic approximation (SPSA) and smoothed functional (SF) methods to estimate gradients and Hessians of the cost-to-go function. The approach combines a model-free, non-parametric policy evaluator (MFMC) with first- and second-order optimization, achieving convergence to optimal policies without function approximators, demonstrated on a 1D continuous control problem.

ABSTRACT

We propose novel policy search algorithms in the context of off-policy, batch mode reinforcement learning (RL) with continuous state and action spaces. Given a batch collection of trajectories, we perform off-line policy evaluation using an algorithm similar to that by [Fonteneau et al., 2010]. Using this Monte-Carlo like policy evaluator, we perform policy search in a class of parameterized policies. We propose both first order policy gradient and second order policy Newton algorithms. All our algorithms incorporate simultaneous perturbation estimates for the gradient as well as the Hessian of the cost-to-go vector, since the latter is unknown and only biased estimates are available. We demonstrate their practicality on a simple 1-dimensional continuous state space problem.

Motivation & Objective

• Address the challenge of off-policy, batch-mode reinforcement learning in continuous state and action spaces without function approximators.
• Develop policy search algorithms that leverage Monte Carlo-like policy evaluation for accurate cost-to-go estimation.
• Introduce first- and second-order optimization methods using simultaneous perturbation to estimate gradients and Hessians when closed-form expressions are unavailable.
• Ensure convergence of policy parameters to near-optimal solutions under biased MFMC estimates.
• Demonstrate practicality on a 1D continuous control problem with theoretical convergence guarantees.

Proposed method

• Propose a non-parametric policy evaluation method (MFMC) that estimates cost-to-go using a batch of trajectories and metrics on state and action spaces.
• Use simultaneous perturbation stochastic approximation (SPSA) and smoothed functional (SF) techniques to estimate the gradient and Hessian of the cost-to-go function.
• Design first-order policy gradient algorithms (MCPG-SPSA, MCPG-SF) that update policy parameters using SPSA or SF gradient estimates.
• Develop second-order Newton-type algorithms (MCPN-SPSA, MCPN-SF) that incorporate Hessian estimates for faster convergence.
• Apply projection-based updates to ensure policy parameters remain within a feasible set.
• Prove almost-sure convergence of the algorithms to the set of asymptotically stable equilibria under mild assumptions, including bounded bias in MFMC estimates.

Experimental results

Research questions

• RQ1 Can simultaneous perturbation methods be effectively applied to estimate gradients and Hessians in off-policy batch RL with unknown dynamics?
• RQ2 How can policy evaluation be performed without function approximators in continuous state and action spaces?
• RQ3 Do first- and second-order SPSA/SF-based policy search algorithms converge to optimal policies under biased MFMC estimates?
• RQ4 What is the theoretical convergence behavior of these algorithms in the infinite-horizon discounted MDP setting?
• RQ5 Can the proposed method achieve practical performance on continuous control problems without relying on value function approximators?

Key findings

• The proposed MCPG-SPSA and MCPG-SF algorithms converge to the set of asymptotically stable equilibria with probability 1 − η, under bounded bias in the MFMC estimator.
• The MCPN-SPSA and MCPN-SF algorithms converge to the set of asymptotically stable equilibria of the Newton update ODE, with probability 1 − η.
• The Hessian estimates in the SPSA-based algorithms converge almost surely to the true Hessian as δ → 0.
• The gradient and Hessian estimators are shown to be consistent in the limit, with bias vanishing asymptotically.
• The method achieves practical performance on a 1D continuous control problem, demonstrating feasibility without function approximators.
• Theoretical convergence is established using ODE-based analysis, with the cost-to-go function serving as a strict Lyapunov function.