[Paper Review] A comparison of sample-based Stochastic Optimal Control methods

Pierre Girardeau|arXiv (Cornell University)|Feb 9, 2010

Advanced Control Systems Optimization13 references4 citations

TL;DR

This paper compares scenario tree and particle-based stochastic optimal control methods using Mean Squared Error (MSE) as a performance metric. While scenario trees suffer from exponential growth in required scenarios with time horizon, the particle method maintains stable convergence independent of time horizon, though performance degrades with increasing state space dimension.

ABSTRACT

In this paper, we compare the performance of two scenario-based numerical methods to solve stochastic optimal control problems: scenario trees and particles. The problem consists in finding strategies to control a dynamicalsystem perturbed by exogenous noises so as to minimize some expected cost along a discrete and finite time horizon. We introduce the Mean SquaredError (MSE) which is the expected $L^2$-distance between the strategy given by the algorithm and the optimal strategy, as a performance indicator for the two models. We study the behaviour of the MSE with respect to the number of scenarios used for discretization. The first model, widely studied in the Stochastic Programming community, consists in approximating the noise diffusion using a scenario tree representation. On a numerical example, we observe that the number of scenarios needed to obtain a given precision growsexponentially with the time horizon. In that sense, our conclusion on scenario trees is equivalent to the one in the work by Shapiro (2006) and has been widely noticed by practitioners. However, in the second part, we show using the same example that, by mixing Stochastic Programming and Dynamic Programmingideas, the particle method described by Carpentier et al. (2009) copes with this numerical difficulty: the number of scenarios needed to obtain a given precision now does not depend on the time horizon. Unfortunately, we alsoobserve that serious obstacles still arise from the system state space dimension.

Motivation & Objective

• To evaluate and compare the performance of scenario tree and particle-based methods in solving stochastic optimal control problems.
• To quantify the impact of time horizon on the number of scenarios required for a given accuracy.
• To assess the influence of state space dimension on the convergence of particle methods.
• To investigate whether particle methods can overcome the exponential scenario growth problem inherent in scenario tree approaches.

Proposed method

• Uses Mean Squared Error (MSE) as the performance indicator, decomposed into bias and variance components.
• Applies scenario trees to discretize noise diffusion over time, treating the problem as a multi-stage stochastic program.
• Employs particle methods that combine Stochastic Programming and Dynamic Programming ideas, using sampling to solve optimality conditions iteratively.
• Constructs adaptive meshes over the state space by concentrating particles in high-probability regions visited by the optimal state.
• Uses nearest-neighbor regression to approximate feedback controls from sampled particles.
• Evaluates convergence rates of bias and variance terms as a function of the number of scenarios N.

Experimental results

Research questions

• RQ1 Does the number of scenarios required for a given accuracy grow exponentially with time horizon in scenario tree methods?
• RQ2 Can particle methods achieve stable convergence independent of time horizon in multi-stage problems?
• RQ3 How does increasing state space dimension affect the convergence rate of particle methods?
• RQ4 Is the variance in particle-based strategies significantly lower than in scenario tree approaches for the same number of scenarios?

Key findings

• Scenario tree methods exhibit a convergence rate of N^(-1/T) with respect to the number of scenarios N, leading to exponential growth in required scenarios as time horizon T increases.
• Particle methods achieve a convergence rate of approximately N^(-1) for bias and variance in one-dimensional state space, independent of time horizon.
• In two-dimensional state space, the convergence rate degrades to approximately N^(-0.5), indicating increased sensitivity to state dimension.
• The variance term in particle methods remains low and stable across time steps, unlike in scenario trees where it grows with horizon.
• The squared bias term dominates in higher-dimensional problems, suggesting limitations in current regression operators like nearest-neighbor.
• Particle methods show promise for multi-stage problems due to time-horizon-invariant convergence, but performance is hindered by state space dimensionality.