[Paper Review] MESA: Offline Meta-RL for Safe Adaptation and Fault Tolerance

Michael Luo, Ashwin Balakrishna+7 more|arXiv (Cornell University)|Dec 7, 2021

Anomaly Detection Techniques and Applications5 citations

TL;DR

MESA proposes an offline meta-RL framework that transfers safety knowledge from diverse offline datasets across environments to rapidly adapt risk measures in new environments with unseen dynamics. By meta-learning a transferable safety critic and fine-tuning it with minimal test data, MESA reduces constraint violations by up to 50% while maintaining task performance in simulation across 5 continuous control domains.

ABSTRACT

Safe exploration is critical for using reinforcement learning (RL) in risk-sensitive environments. Recent work learns risk measures which measure the probability of violating constraints, which can then be used to enable safety. However, learning such risk measures requires significant interaction with the environment, resulting in excessive constraint violations during learning. Furthermore, these measures are not easily transferable to new environments. We cast safe exploration as an offline meta-RL problem, where the objective is to leverage examples of safe and unsafe behavior across a range of environments to quickly adapt learned risk measures to a new environment with previously unseen dynamics. We then propose MEta-learning for Safe Adaptation (MESA), an approach for meta-learning a risk measure for safe RL. Simulation experiments across 5 continuous control domains suggest that MESA can leverage offline data from a range of different environments to reduce constraint violations in unseen environments by up to a factor of 2 while maintaining task performance. See https://tinyurl.com/safe-meta-rl for code and supplementary material.

Motivation & Objective

• To address the challenge of unsafe exploration in risk-sensitive RL environments where online interaction risks costly failures.
• To enable safe adaptation in new environments with previously unseen dynamics, such as partial actuator failures or changing system parameters.
• To reduce reliance on online interaction for learning risk measures by leveraging offline datasets from diverse training environments.
• To develop a method that transfers safety knowledge across environments without requiring task-specific data in the test phase.
• To maintain high task performance while minimizing constraint violations during adaptation in unseen environments.

Proposed method

• MESA formulates safe RL as an offline meta-RL problem, where meta-training occurs on offline datasets from multiple training environments with varying dynamics.
• It meta-learns a safety critic $Q^{ ext{risk}}_{\pi}$ that estimates the risk of constraint violations across diverse environments.
• In the adaptation phase, the meta-learned safety critic is fine-tuned using a small, offline test dataset from a new environment with previously unseen dynamics.
• The adapted safety critic is then used in conjunction with a recovery policy to enable safe online learning in the test environment, following the Recovery RL paradigm.
• The method relies on a contrastive learning objective to align risk estimates across environments and improve generalization to out-of-distribution dynamics.
• It uses a multi-head network architecture to jointly predict task return and risk, enabling end-to-end training with shared representations.

Experimental results

Research questions

• RQ1 Can offline meta-RL transfer safety knowledge from diverse environments to enable rapid adaptation in new environments with unseen dynamics?
• RQ2 How small can the test dataset be while still enabling effective safety adaptation with minimal constraint violations?
• RQ3 How well does MESA generalize to environments with significantly different dynamics, such as partial joint failures, compared to baseline methods?
• RQ4 To what extent does MESA maintain task performance while reducing constraint violations during adaptation?
• RQ5 Can MESA achieve safe adaptation without requiring online interaction or task-specific data in the test environment?

Key findings

• MESA reduces constraint violations by up to a factor of 2 in unseen environments compared to prior offline RL methods, even with minimal test data.
• The method maintains strong task performance while significantly outperforming multi-task learning baselines in terms of both safety and sample efficiency.
• MESA generalizes effectively to environments with partial joint failures, achieving superior performance and fewer constraint violations than Multi-Task learning.
• Performance degrades only when the test dataset is reduced to 1/16th of the original size (10K transitions), indicating robustness to small test data.
• With a test dataset of 1/4 the original size (10K transitions), MESA maintains high performance, suggesting it can operate with up to 40x less test data than training data.
• Ablation studies confirm that MESA's meta-learned risk measure generalizes well to out-of-distribution dynamics, such as partial actuator failures not seen during meta-training.