[Paper Review] A numerical approach to stochastic reach-avoid problems for Markov Decision Processes.

Nikolaos Kariotoglou, Maryam Kamgarpour+2 more|arXiv (Cornell University)|Nov 21, 2014

Advanced Control Systems Optimization50 references3 citations

TL;DR

This paper proposes a numerical method for solving high-dimensional stochastic reach-avoid problems in Markov Decision Processes by relaxing recursive equations into inequalities, projecting the value function onto Gaussian radial basis functions, and sampling constraints. The approach enables analytical computation of one-step rewards for hyper-rectangular safe and target sets, significantly outperforming grid-based methods and demonstrating effectiveness on a nonlinear autonomous race car path planning problem.

ABSTRACT

We consider finite horizon reach-avoid problems for discrete time stochastic systems with additive Gaussian mixture noise. Our goal is to approximate the optimal value function of such problems on dimensions higher than what can be handled via state-space gridding techniques. We achieve this by relaxing the recursive equations of the finite horizon reach-avoid optimal control problem into inequalities, projecting the optimal value function to a finite dimensional basis and sampling the associated infinite set of constraints. We focus on a specific value function parametrization using Gaussian radial basis functions that enables the analytical computation of the one-step reach-avoid reward in the case of hyper-rectangular safe and target sets, achieving significant computational benefits compared to state-space gridding. We analyze the performance of the overall method numerically by approximating simple reach-avoid control problems and comparing the results to benchmark controllers based on well-studied methods. The full power of the method is demonstrated on a nonlinear control problem inspired from constrained path planning for autonomous race cars.

Motivation & Objective

• To address the challenge of solving finite horizon reach-avoid problems in high-dimensional stochastic systems where state-space gridding becomes computationally infeasible.
• To develop a scalable approximation method for the optimal value function in systems with additive Gaussian mixture noise.
• To enable analytical computation of one-step reach-avoid rewards through a specific parametrization using Gaussian radial basis functions.
• To demonstrate the method's performance on both simple benchmark problems and a complex nonlinear autonomous vehicle control problem.
• To provide a computationally efficient alternative to traditional grid-based methods for stochastic optimal control.

Proposed method

• Relax the recursive equations of the finite horizon reach-avoid optimal control problem into inequalities to enable numerical approximation.
• Project the optimal value function onto a finite-dimensional basis of Gaussian radial basis functions (RBFs) for tractable computation.
• Sample an infinite set of constraints arising from the relaxed inequalities to form a finite optimization problem.
• Leverage the structure of hyper-rectangular safe and target sets to analytically compute one-step reach-avoid rewards, reducing reliance on numerical integration.
• Formulate the problem as a constrained optimization task that can be solved using standard numerical solvers.
• Validate the method on low-dimensional problems and scale it to a high-dimensional nonlinear path planning problem for autonomous race cars.

Experimental results

Research questions

• RQ1 Can a relaxation-based approach with function approximation outperform traditional state-space gridding in high-dimensional stochastic reach-avoid problems?
• RQ2 To what extent can Gaussian radial basis functions enable analytical computation of one-step rewards in reach-avoid settings with hyper-rectangular sets?
• RQ3 How does the proposed method compare to benchmark controllers in terms of performance and computational efficiency?
• RQ4 Can the method be effectively scaled to nonlinear, high-dimensional systems such as autonomous vehicle path planning?
• RQ5 What is the impact of constraint sampling and basis function selection on the accuracy and convergence of the value function approximation?

Key findings

• The method achieves significant computational advantages over state-space gridding by enabling analytical computation of one-step rewards through Gaussian RBF parametrization.
• Numerical results show that the proposed approach approximates the optimal value function with high accuracy on simple reach-avoid problems, outperforming benchmark controllers.
• The method successfully solves a challenging nonlinear path planning problem for autonomous race cars, demonstrating scalability and practical relevance.
• The use of Gaussian radial basis functions allows for efficient and accurate projection of the value function in high-dimensional spaces.
• Constraint sampling effectively reduces the infinite-dimensional problem to a finite, solvable optimization problem without sacrificing solution quality.
• The analytical treatment of the one-step reward for hyper-rectangular sets eliminates costly numerical integration, enhancing computational efficiency.