[Paper Review] Measuring the Reliability of Reinforcement Learning Algorithms

Stephanie C. Y. Chan, Samuel H. Fishman+3 more|arXiv (Cornell University)|Dec 10, 2019

Reinforcement Learning in Robotics29 references41 citations

TL;DR

The paper defines a set of open-source reliability metrics for RL that quantify dispersion and risk across time, runs, and fixed-policy rollouts, along with non-parametric statistical tests for comparisons.

ABSTRACT

Lack of reliability is a well-known issue for reinforcement learning (RL) algorithms. This problem has gained increasing attention in recent years, and efforts to improve it have grown substantially. To aid RL researchers and production users with the evaluation and improvement of reliability, we propose a set of metrics that quantitatively measure different aspects of reliability. In this work, we focus on variability and risk, both during training and after learning (on a fixed policy). We designed these metrics to be general-purpose, and we also designed complementary statistical tests to enable rigorous comparisons on these metrics. In this paper, we first describe the desired properties of the metrics and their design, the aspects of reliability that they measure, and their applicability to different scenarios. We then describe the statistical tests and make additional practical recommendations for reporting results. The metrics and accompanying statistical tools have been made available as an open-source library at https://github.com/google-research/rl-reliability-metrics. We apply our metrics to a set of common RL algorithms and environments, compare them, and analyze the results.

Motivation & Objective

• Motivate the need to quantify reliability in RL due to high variability across seeds, environments, and implementations.
• Define general-purpose reliability metrics that capture multiple aspects of variability during training and after learning.
• Distinguish between dispersion and risk, and between evaluation during training and evaluation after learning.
• Provide statistical tools and reporting recommendations to enable rigorous, comparable reliability analyses.

Proposed method

• Define three axes of variability: Across Time during training, Across Runs during training, and Across rollouts of a fixed policy after learning.
• Use two measures of variability for each axis: dispersion (robust statistics like IQR) and risk (CVaR) to capture heavy tails.
• Compute Dispersion across Time by detrending the training curve and applying IQR within sliding windows.
• Compute Short-term Risk across Time as CVaR on first-order differences of performance.
• Compute Long-term Risk across Time as CVaR on Drawdown relative to the running peak.
• Compute Dispersion across Runs as IQR across low-pass-filtered run performances; compute Risk across Runs as CVaR across runs.
• Compute Dispersion across Fixed-Policy Rollouts as IQR across rollouts of a fixed policy; compute Risk across Fixed-Policy Rollouts as CVaR across those rollouts.
• Address invariance to evaluation frequency and use bootstrap confidence intervals and permutation tests for comparisons.
• Recommend reporting all metrics with clearly specified parameters (window size, filtering, evaluation frequency, etc.) and normalization to enable fair comparisons.

Experimental results

Research questions

• RQ1 How can reliability in RL be quantified beyond mean/median performance across environments and seeds?
• RQ2 What are robust statistical tools to compare reliability metrics across algorithms and environments?
• RQ3 Do reliability patterns align with, or diverge from, median performance, and can environment-specific analyses reveal strengths/weaknesses?
• RQ4 How should results be reported to enable fair, reproducible comparisons across studies?

Key findings

• Reliability metrics often reveal differences not captured by median performance; algorithms with strong mean performance can have poor reliability and vice versa.
• In continuous control experiments, SAC and TD3 show strong reliability during training but may underperform on reliability after learning compared to their training performance.
• In Atari experiments, Rainbow may have better median performance but IQN can show stronger reliability on several metrics.
• Reliability patterns can vary by environment, underscoring the value of per-environment analysis alongside aggregated results.
• An open-source Python package accompanies the work to facilitate adoption of the metrics and statistical tools.