[Paper Review] Rigorous Agent Evaluation: An Adversarial Approach to Uncover Catastrophic Failures

Jonathan Uesato, Ananya Kumar+8 more|arXiv (Cornell University)|Dec 4, 2018

Adversarial Robustness in Machine Learning43 references46 citations

TL;DR

The paper introduces adversarial evaluation for RL agents to efficiently find and estimate catastrophic failures, outperforming vanilla Monte Carlo by leveraging failure probability predictors learned from weaker agents.

ABSTRACT

This paper addresses the problem of evaluating learning systems in safety critical domains such as autonomous driving, where failures can have catastrophic consequences. We focus on two problems: searching for scenarios when learned agents fail and assessing their probability of failure. The standard method for agent evaluation in reinforcement learning, Vanilla Monte Carlo, can miss failures entirely, leading to the deployment of unsafe agents. We demonstrate this is an issue for current agents, where even matching the compute used for training is sometimes insufficient for evaluation. To address this shortcoming, we draw upon the rare event probability estimation literature and propose an adversarial evaluation approach. Our approach focuses evaluation on adversarially chosen situations, while still providing unbiased estimates of failure probabilities. The key difficulty is in identifying these adversarial situations -- since failures are rare there is little signal to drive optimization. To solve this we propose a continuation approach that learns failure modes in related but less robust agents. Our approach also allows reuse of data already collected for training the agent. We demonstrate the efficacy of adversarial evaluation on two standard domains: humanoid control and simulated driving. Experimental results show that our methods can find catastrophic failures and estimate failures rates of agents multiple orders of magnitude faster than standard evaluation schemes, in minutes to hours rather than days.

Motivation & Objective

• Motivate reliable evaluation of learning systems in safety-critical domains where failures are catastrophic (e.g., autonomous driving).
• Show limitations of vanilla random testing in detecting rare failures and estimating risk.
• Propose an adversarial evaluation framework that uses a failure probability predictor (AVF) learned from weaker agents to guide failure search and risk estimation.
• Demonstrate that adversarial evaluation finds failures and estimates failure probabilities orders of magnitude faster than standard methods.

Proposed method

• Define a failure indicator c(x,Z) for agent given initial condition x and randomness Z.
• Introduce AVF (failure probability predictor f*(x)=P(c(x,Z)=1)) and a continuation approach to learn f ≈ f* from related, weaker agents.
• Use AVF to guide failure search by selecting high-fx initial conditions, with diversity to improve robustness.
• Apply AVF-guided importance sampling for risk estimation by constructing a proposal distribution Q_f to minimize estimator variance.
• Provide Algorithm 1: AVF-guided risk estimator (AVF estimator) which samples from P_X with acceptance probability f^α(X_t) and weights outcomes by f^-α(X_t).
• Describe a continuation strategy where AVF is learned from agents earlier in training to provide stronger signals for evaluation.

Experimental results

Research questions

• RQ1 Can adversarial evaluation uncover catastrophic failures more efficiently than vanilla Monte Carlo in RL agents?
• RQ2 How can a failure probability predictor (AVF) be learned from weaker, related agents to guide failure search and risk estimation?
• RQ3 How much data and how many environment interactions are saved when using AVF-guided evaluation compared to standard methods?
• RQ4 Does AVF-based risk estimation maintain unbiasedness while reducing variance via importance sampling?
• RQ5 Can AVF-based methods aid in identifying the most reliable agent from a finite set?

Key findings

• AVF adversaries find adversarial inputs with substantially fewer episodes than random testing (e.g., Driving: 198x fewer; Humanoid: 3100x).
• AVF-guided risk estimation drastically reduces the number of experiments needed to achieve a given accuracy (Driving: 750 vs 11,000 trials for 3-approximation; Humanoid: 15,000 vs 5.1e5 trials).
• AVF-based approach provides orders of magnitude faster and more robust failure search and risk estimation, enabling practical reliability assessment within minutes to hours rather than days.
• Prioritized Replay (PR) adversaries improve efficiency but may miss some failures, requiring fallback to VMC in certain cases.
• Using AVF for model selection can better rank policies by reliability compared to VMC, identifying more robust agents earlier in training.