[Paper Review] Challenges of Real-World Reinforcement Learning

Gabriel Dulac-Arnold, Daniel J. Mankowitz+1 more|arXiv (Cornell University)|Apr 29, 2019

Reinforcement Learning in Robotics67 references252 citations

TL;DR

This paper identifies nine practical challenges for deploying RL in real-world systems, surveys existing approaches and evaluation metrics for each, and demonstrates them using a modified control suite as a testbed.

ABSTRACT

Reinforcement learning (RL) has proven its worth in a series of artificial domains, and is beginning to show some successes in real-world scenarios. However, much of the research advances in RL are often hard to leverage in real-world systems due to a series of assumptions that are rarely satisfied in practice. We present a set of nine unique challenges that must be addressed to productionize RL to real world problems. For each of these challenges, we specify the exact meaning of the challenge, present some approaches from the literature, and specify some metrics for evaluating that challenge. An approach that addresses all nine challenges would be applicable to a large number of real world problems. We also present an example domain that has been modified to present these challenges as a testbed for practical RL research.

Motivation & Objective

• Motivate the need for RL methods that work in real-world, data-limited, safety-constrained settings.
• Characterize nine concrete challenges that differentiate real-world RL from laboratory RL.
• Propose evaluation frameworks and metrics tailored to each challenge.
• Illustrate the challenges and evaluation strategies on a modified DeepMind control suite humanoid task.

Proposed method

• Define and formalize nine real-world RL challenges within an MDP/POMDP/safety-constrained context.
• Survey literature-aligned approaches for each challenge (batch/off-policy training, sample efficiency, high-dimensional spaces, safety constraints, partial observability/non-stationarity, unspecified/multi-objective rewards, explainability, real-time inference, and delays).
• Propose concrete evaluation metrics per challenge (e.g., warm-start performance, data efficiency, safety violation counts, worst-case performance, CVaR, multi-objective vectors).
• Present a testbed by modifying a control suite task to incorporate all challenges, plus guidance for evaluating each aspect in practice.

Experimental results

Research questions

• RQ1 What are the essential challenges to productionize RL in real-world systems?
• RQ2 How can each challenge be defined, measured, and evaluated?
• RQ3 What existing methods address these challenges, and what gaps remain when considering them jointly?
• RQ4 Can a testbed demonstrate simultaneous handling of all nine challenges?

Key findings

• A comprehensive set of nine real-world RL challenges is proposed, each with definitions, literature approaches, and evaluation metrics.
• An integrated evaluation framework is offered, emphasizing safety, robustness, and multi-objective considerations over sole average returns.
• Batch/off-policy training, data efficiency, safety via CMDPs, and real-time inference emerge as critical for production viability.
• A toy-example within a modified DeepMind control suite demonstrates how to stress-test algorithms across all nine challenges.
• The paper argues that algorithms addressing all nine challenges would be applicable to a broad class of real-world problems.