[Paper Review] An empirical investigation of the challenges of real-world reinforcement learning

Gabriel Dulac-Arnold, Nir Levine+5 more|arXiv (Cornell University)|Mar 24, 2020

Reinforcement Learning in Robotics133 references52 citations

TL;DR

The paper formalizes nine real-world RL challenges, analyzes their effects on SOTA agents using realworldrl-suite, and proposes an open-source benchmark for evaluation.

ABSTRACT

Reinforcement learning (RL) has proven its worth in a series of artificial domains, and is beginning to show some successes in real-world scenarios. However, much of the research advances in RL are hard to leverage in real-world systems due to a series of assumptions that are rarely satisfied in practice. In this work, we identify and formalize a series of independent challenges that embody the difficulties that must be addressed for RL to be commonly deployed in real-world systems. For each challenge, we define it formally in the context of a Markov Decision Process, analyze the effects of the challenge on state-of-the-art learning algorithms, and present some existing attempts at tackling it. We believe that an approach that addresses our set of proposed challenges would be readily deployable in a large number of real world problems. Our proposed challenges are implemented in a suite of continuous control environments called the realworldrl-suite which we propose an as an open-source benchmark.

Motivation & Objective

• Identify and define real-world RL challenges and their intuition in MDPS.
• Provide formal definitions and analyze the impact of each challenge on learning algorithms.
• Develop a benchmark suite (realworldrl-suite) extending the DeepMind Control Suite to study challenges.
• Evaluate state-of-the-art agents (DMPO and D4PG) across challenges to establish baselines.
• Offer guidance and resources to enable reproducible testing of RL in real-world-like settings.

Proposed method

• Formally define nine real-world RL challenges within the MDP framework.
• Implement challenging environments in realworldrl-suite, extending the DeepMind Control Suite with perturbations.
• Benchmark two SOTA agents (DMPO and D4PG) on multiple tasks with varying difficulty.
• Introduce pre-convergence regret and post-convergence instability metrics to assess sample efficiency and stability.
• Calibrate and combine a subset of challenges into a combined benchmark task for baseline comparison.
• Provide open-source code and documentation for reproducing experiments.

Experimental results

Research questions

• RQ1 How does each real-world challenge affect RL learning performance and sample efficiency?
• RQ2 How do DMPO and D4PG compare under these real-world challenges?
• RQ3 What is the impact of combining challenges into a single benchmark task?
• RQ4 Which challenges are most detrimental to stability and convergence across continuous control tasks?

Key findings

• DMPO exhibits higher pre-convergence regret than D4PG across all tasks.
• D4PG generally demonstrates greater sample efficiency and, in many cases, more stable convergence than DMPO.
• Increasing delays in actions, observations, or rewards degrades performance, with action/observation delays being particularly impactful.
• Adding high-dimensional or noisy dummy state dimensions can slow convergence but Learners can still reach near-optimal performance on some tasks.
• A combined real-world challenge benchmark reveals that state-of-the-art agents can fail quickly under mild perturbations, highlighting the need for more robust methods.
• The paper provides an open-source benchmark (realworldrl-suite) to standardize evaluation of these challenges.