[Paper Review] Conformal Prediction Intervals for Markov Decision Process Trajectories

Thomas G. Dietterich, Jesse Hostetler|arXiv (Cornell University)|Jun 10, 2022

Simulation Techniques and Applications6 citations

TL;DR

This paper proposes SQBoxCI, a conformal prediction method that generates trajectory-wise prediction intervals for Markov Decision Process (MDP) trajectories, ensuring finite-sample coverage of $1 - \delta$ across all time steps. By combining quantile regression with conformal correction, it produces tighter, joint multivariate intervals than traditional Bonferroni-based methods, validated on invasive species management and StarCraft2 tasks with strong empirical coverage.

ABSTRACT

Before delegating a task to an autonomous system, a human operator may want a guarantee about the behavior of the system. This paper extends previous work on conformal prediction for functional data and conformalized quantile regression to provide conformal prediction intervals over the future behavior of an autonomous system executing a fixed control policy on a Markov Decision Process (MDP). The prediction intervals are constructed by applying conformal corrections to prediction intervals computed by quantile regression. The resulting intervals guarantee that with probability $1-δ$ the observed trajectory will lie inside the prediction interval, where the probability is computed with respect to the starting state distribution and the stochasticity of the MDP. The method is illustrated on MDPs for invasive species management and StarCraft2 battles.

Motivation & Objective

• To provide finite-sample, $1-\delta$ coverage guarantees for multivariate prediction intervals over entire MDP trajectories, not just individual time steps.
• To address the limitations of Bonferroni correction in multivariate prediction by jointly modeling trajectory behavior, resulting in tighter and more informative intervals.
• To develop a method that is robust to extreme quantile estimation and provides interpretable, safety-critical confidence bounds for autonomous system deployment.
• To enable early warning of potential prediction failures by analyzing violation patterns in training data, supporting human-AI collaboration in safety-critical domains.

Proposed method

• Uses quantile regression to predict $\delta/2$ and $1-\delta/2$ quantiles of the behavior vector at each time step as a function of the starting state $s_0$.
• Applies the SBox algorithm to compute multivariate conformal prediction intervals by rescaling and conformalizing the quantile regression outputs across time steps and dimensions.
• Introduces SQBoxCI, a two-stage method that first computes per-time-step quantile predictions and then applies conformal correction to ensure joint trajectory-wise coverage.
• Employs split-conformal prediction to ensure finite-sample validity, with the guarantee holding over the distribution of starting states $P_0$.
• Uses exceedance measures to assess how often the true trajectory lies outside the predicted interval, with the max exceedance used to calibrate the final bounds.
• Incorporates heuristic checks to detect clustering of violations in state space, improving interpretability and reducing false positives in safety-critical settings.

Experimental results

Research questions

• RQ1 Can conformal prediction be effectively extended to multivariate, trajectory-wise prediction intervals in MDPs while maintaining finite-sample coverage?
• RQ2 How does the joint conformal approach compare to Bonferroni-corrected univariate intervals in terms of interval tightness and empirical coverage?
• RQ3 Can the method detect and flag regions of the state space where prediction intervals are systematically violated, enabling early warning for human operators?
• RQ4 How robust is the method to small training sample sizes and extreme quantile estimation in high-dimensional trajectory spaces?

Key findings

• The SQBoxCI method achieves empirical coverage close to the nominal $1 - \delta$ level across both the Tamarisk invasive species management and StarCraft2 battle tasks.
• For $\delta = 0.2$ and $\delta = 0.1$, the total exceedance-based bound method produced reasonably faithful depictions of future behavior, though performance degraded for $\delta = 0.05$ in the Tamarisk task.
• The conformal correction produced significantly tighter prediction intervals than Bonferroni-corrected univariate intervals, especially in high-dimensional settings.
• Experiments showed that at least 500 trajectories are needed for stable and tight conformal quantile estimates, highlighting a data efficiency limitation.
• Violation clustering was detected in the StarCraft2 task, indicating that some starting states consistently led to interval violations, which the method can help identify through training data inspection.
• The method enables interpretable, semi-conditional prediction intervals that, when violations are well-distributed, behave nearly as if conditionally valid for individual starting states.