[Paper Review] Sampling-based Approximations with Quantitative Performance for the Probabilistic Reach-Avoid Problem over General Markov Processes

Sofie Haesaert, Robert Babuška+1 more|arXiv (Cornell University)|Sep 1, 2014

Formal Methods in Verification25 references3 citations

TL;DR

This paper proposes a sampling-based Fitted Value Iteration (FVI) algorithm for computing optimal controllers in continuous-state Markov processes with control inputs, targeting probabilistic reach-avoid specifications. It provides a priori computable probabilistic error bounds on the approximation, enabling rigorous performance quantification for safety-critical applications.

ABSTRACT

This article deals with stochastic processes endowed with the Markov (memoryless) property and evolving over general (uncountable) state spaces. The models further depend on a non-deterministic quantity in the form of a control input, which can be selected to affect the probabilistic dynamics. We address the computation of maximal reach-avoid specifications, together with the synthesis of the corresponding optimal controllers. The reach-avoid specification deals with assessing the likelihood that any finite-horizon trajectory of the model enters a given goal set, while avoiding a given set of undesired states. This article newly provides an approximate computational scheme for the reach-avoid specification based on the Fitted Value Iteration algorithm, which hinges on random sample extractions, and gives a-priori computable formal probabilistic bounds on the error made by the approximation algorithm: as such, the output of the numerical scheme is quantitatively assessed and thus meaningful for safety-critical applications. Furthermore, we provide tighter probabilistic error bounds that are sample-based. The overall computational scheme is put in relationship with alternative approximation algorithms in the literature, and finally its performance is practically assessed over a benchmark case study.

Motivation & Objective

• Address the challenge of synthesizing optimal controllers for probabilistic reach-avoid specifications in general continuous-state Markov processes with control inputs.
• Overcome the limitations of existing approximation schemes that lack formal error bounds by providing computable probabilistic error bounds on the FVI approximation.
• Enable practical deployment of controller synthesis in safety-critical systems by quantifying the accuracy of the computed policy and value function.
• Extend the applicability of dynamic programming to uncountable state spaces with hybrid dynamics, using sample-based regression.
• Provide tighter, sample-based error bounds that improve upon asymptotic convergence guarantees in prior methods.

Proposed method

• Apply Fitted Value Iteration (FVI) to approximate the value function of the reach-avoid problem in continuous state spaces using random samples.
• Use kernel-based regression (specifically, ClassificationTree.fit in MATLAB) to fit the value function at each iteration, replacing exact dynamic programming steps.
• Derive a priori probabilistic error bounds on the FVI approximation using measure-theoretic analysis and concentration inequalities.
• Introduce sample-based error estimates for the single-step error and bias in the dynamic programming operator, computed via hold-out samples.
• Propagate error bounds backward over time horizons using Theorem 12, which quantifies cumulative approximation error via a scaling factor B.
• Leverage the FVI framework to compute a near-optimal policy μ̂* and value function Ŵ* with formal performance guarantees.

Experimental results

Research questions

• RQ1 Can a sampling-based FVI algorithm provide formal, a priori probabilistic bounds on the error in approximating the reach-avoid probability for continuous-state Markov processes?
• RQ2 How does the error in the FVI approximation propagate over time, and can this be bounded using computable metrics?
• RQ3 What is the impact of sample size and sampling distribution on the accuracy of the FVI approximation in the reach-avoid problem?
• RQ4 Can tighter, sample-based error estimates be derived that improve upon asymptotic convergence results in existing literature?
• RQ5 How does the performance of the FVI-based controller compare to exact dynamic programming in a realistic benchmark case study?

Key findings

• The FVI algorithm achieves a single-step error estimate between 3×10⁻³ and 5×10⁻³ across all iterations, with a noticeable trend of decreasing error in earlier iterations.
• The cumulative approximation error grows exponentially over the time horizon, with a computed scaling factor B = 3.07, indicating strong error propagation.
• The sample-based accuracy estimate Δ for 1−δΔ=0.9 remains below 10⁻² for early iterations but increases significantly toward the end of the horizon.
• The policy μ̂* derived via FVI shows logical behavior in the benchmark case study: heaters are kept on longer in the room with higher heat loss, consistent with physical intuition.
• The FVI-based value function Ŵ* captures the reach-avoid structure accurately in regions near the goal set K, though performance degrades in flat regions far from K.
• The proposed method provides formal, a priori probabilistic error bounds, distinguishing it from prior schemes that only offer asymptotic convergence guarantees.