[Paper Review] Out-of-Distribution Dynamics Detection: RL-Relevant Benchmarks and Results

Mohamad H. Danesh, Alan Fern|arXiv (Cornell University)|Jul 11, 2021

Anomaly Detection Techniques and Applications25 references4 citations

TL;DR

This paper introduces the first benchmark suite for out-of-distribution dynamics detection (OODD) in reinforcement learning environments, derived from standard RL benchmarks like OpenAI Gym and Bullet. It proposes a recurrent implicit quantile network (RIQN)-based detection method that identifies anomalies via prediction error in autoregressive models, achieving non-trivial performance with room for improvement on AUC, detection delay, and false alarm rate metrics.

ABSTRACT

We study the problem of out-of-distribution dynamics (OODD) detection, which involves detecting when the dynamics of a temporal process change compared to the training-distribution dynamics. This is relevant to applications in control, reinforcement learning (RL), and multi-variate time-series, where changes to test time dynamics can impact the performance of learning controllers/predictors in unknown ways. This problem is particularly important in the context of deep RL, where learned controllers often overfit to the training environment. Currently, however, there is a lack of established OODD benchmarks for the types of environments commonly used in RL research. Our first contribution is to design a set of OODD benchmarks derived from common RL environments with varying types and intensities of OODD. Our second contribution is to design a strong OODD baseline approach based on recurrent implicit quantile network (RIQN), which monitors autoregressive prediction errors for OODD detection. In addition to RIQN, we introduce and test three other simpler baselines. Our final contribution is to evaluate our baseline approaches on the benchmarks to provide results for future comparison.

Motivation & Objective

• To address the lack of standardized benchmarks for out-of-distribution dynamics detection (OODD) in reinforcement learning environments.
• To create realistic OODD benchmarks by injecting diverse dynamic anomalies into common RL environments such as CartPole, LunarLander, and Ant.
• To develop and evaluate baseline OODD detection methods based on predictive failure using autoregressive models, including RIQN, RNNs, random forests, and deterministic models.
• To provide a reproducible evaluation framework with metrics including AUC, detection delay, and false alarm rate for future method comparison.
• To inspire future research by releasing public code for benchmarks and baselines on GitHub.

Proposed method

• The benchmark environments are constructed by training nominal policies using IQN (for OpenAI Gym) and TD3 (for Bullet) to generate 10,000 nominal trajectories per environment.
• Anomalous dynamics are injected into test-time trajectories by modifying environment parameters (e.g., mass, friction, gravity) at random time steps, creating 1,000 anomalous trajectories per environment.
• A recurrent implicit quantile network (RIQN) is used as the core predictive model to estimate the distribution of future observations based on past states.
• Anomaly scores are computed as the distance between the true observation and the predicted distribution (e.g., via quantile-based loss), with larger distances indicating higher anomaly likelihood.
• Three simpler baselines are evaluated: random forests, deterministic RNNs, and a standard RNN-based predictor, all using prediction error as the anomaly signal.
• Evaluation uses three metrics: AUC (discriminative power), detection delay (time to flag anomaly), and false alarm rate (spurious detections).

Experimental results

Research questions

• RQ1 How effective are different predictive models in detecting out-of-distribution dynamics changes in standard RL environments?
• RQ2 What is the trade-off between detection delay, AUC, and false alarm rate across various anomaly types and intensities?
• RQ3 Can a probabilistic, autoregressive model like RIQN outperform simpler baselines in OODD detection under high-dimensional state spaces?
• RQ4 How do different anomaly injection strategies (e.g., mass changes, friction shifts) affect detectability and model performance?
• RQ5 To what extent do current baselines achieve reliable OODD detection, and what gaps remain for future method development?

Key findings

• The RIQN-based baseline achieves non-trivial performance on the OODD benchmarks, demonstrating the feasibility of using predictive uncertainty for anomaly detection in RL dynamics.
• Detection performance varies significantly across environments and anomaly types, with higher AUC values observed in environments like Ant and Hopper compared to simpler ones like CartPole.
• The detection delay is generally low for RIQN, indicating timely anomaly response, though it increases with higher anomaly intensity and complexity.
• False alarm rates are relatively low for RIQN and deterministic RNNs, but random forest baselines show higher false positives, especially in high-dimensional state spaces.
• The benchmarks reveal that current methods still have substantial room for improvement, particularly in balancing low delay with high AUC and low false alarm rates.
• The public release of benchmarks and code enables reproducible evaluation and encourages future research in OODD detection for RL.