[Paper Review] Risk Sensitive Dead-end Identification in Safety-Critical Offline Reinforcement Learning

Taylor W. Killian, Sonali Parbhoo+1 more|arXiv (Cornell University)|Jan 13, 2023

Adversarial Robustness in Machine Learning4 citations

TL;DR

This paper proposes Distributional Dead-end Discovery (DistDeD), a risk-sensitive framework for identifying worst-case outcomes in safety-critical offline reinforcement learning by modeling the full distribution of expected returns, enabling earlier and more tunable detection of high-risk decision points. DistDeD improves over prior methods by detecting dead-ends 10 hours earlier on average and increasing detection by 20% in ICU patient risk assessment.

ABSTRACT

In safety-critical decision-making scenarios being able to identify worst-case outcomes, or dead-ends is crucial in order to develop safe and reliable policies in practice. These situations are typically rife with uncertainty due to unknown or stochastic characteristics of the environment as well as limited offline training data. As a result, the value of a decision at any time point should be based on the distribution of its anticipated effects. We propose a framework to identify worst-case decision points, by explicitly estimating distributions of the expected return of a decision. These estimates enable earlier indication of dead-ends in a manner that is tunable based on the risk tolerance of the designed task. We demonstrate the utility of Distributional Dead-end Discovery (DistDeD) in a toy domain as well as when assessing the risk of severely ill patients in the intensive care unit reaching a point where death is unavoidable. We find that DistDeD significantly improves over prior discovery approaches, providing indications of the risk 10 hours earlier on average as well as increasing detection by 20%.

Motivation & Objective

• To address the challenge of detecting worst-case outcomes in safety-critical offline reinforcement learning where exploration is infeasible and data is limited.
• To overcome the limitations of prior dead-end discovery (DeD) methods that rely on point estimates of expected return, which can be overly optimistic and fail to capture risk distribution.
• To develop a framework that provides earlier, tunable, and conservative risk indicators by modeling the full distribution of return outcomes.
• To enable human operators to intervene proactively in high-risk scenarios before irreversible decline occurs, especially in clinical and industrial settings.

Proposed method

• The method models the full distribution of expected returns using distributional reinforcement learning (DrQ), replacing point estimates with learned return distributions for each action in a given state.
• It computes a risk-sensitive measure based on the lower tail of the return distribution, specifically the worst-case outcomes, to identify states at risk of leading to dead-ends.
• The framework introduces a tunable risk threshold that allows adjustment based on task-specific risk tolerance, enabling conservative or aggressive detection strategies.
• It establishes theoretical grounding by showing that the risk estimate is a lower-bound to the original DeD method, preserving its core safety guarantees while improving sensitivity.
• The approach is applied to both a toy environment and a real-world ICU dataset, using offline datasets to train and evaluate the model without online interaction.
• Dead-end detection is performed by evaluating whether the worst-case return distribution for any action falls below a safety threshold, signaling imminent system failure or patient deterioration.

Figure 1 : Illustration of the determination of conditional value at risk ( $\mathrm{CVaR}_{\alpha}$ ), with $\alpha=0.1$

Experimental results

Research questions

• RQ1 Can modeling the full distribution of expected returns improve early detection of worst-case outcomes in safety-critical offline RL?
• RQ2 How does distributional risk estimation compare to point-estimate-based methods in detecting dead-ends before irreversible decline?
• RQ3 To what extent can the framework be tuned to different risk tolerances while maintaining reliability in high-stakes environments?
• RQ4 Can DistDeD detect impending system failure or patient deterioration earlier than existing methods in real-world clinical data?

Key findings

• DistDeD detects high-risk scenarios 10 hours earlier on average compared to prior dead-end discovery methods in ICU patient monitoring.
• The method increases detection of impending dead-ends by 20% compared to baseline approaches, improving sensitivity to worst-case outcomes.
• By modeling the full return distribution, DistDeD provides a tunable risk-sensitivity framework that can be customized to clinical or industrial safety requirements.
• The risk estimate derived from the distributional approach serves as a theoretical lower-bound to the original DeD method, ensuring safety guarantees are preserved.
• In both synthetic and real-world ICU data, DistDeD successfully identifies states where patient outcomes are likely to deteriorate irreversibly, even when data is limited and confounded.
• The framework demonstrates practical utility as a decision-support tool, enabling earlier human intervention in safety-critical systems without requiring online exploration.

Figure 2 : Distributional Dead-end Discovery (DistDeD) a) Observations are encoded (as needed) into a state representation and then b) passed to independent IQN models to estimate the distribution of returns ( $Z_{D}$ and $Z_{R}$ ) for each possible action. c) The $\mathrm{CVaR}_{\alpha}$ is compute