[Paper Review] Evaluating the progress of Deep Reinforcement Learning in the real world: aligning domain-agnostic and domain-specific research

Juan Jose Garau Luis, Edward F. Crawley+1 more|arXiv (Cornell University)|Jul 7, 2021

Reinforcement Learning in Robotics78 references4 citations

TL;DR

This paper evaluates the gap between domain-agnostic and domain-specific Deep Reinforcement Learning (DRL) research by identifying five key shortcomings in general DRL research—such as bias toward robotics, lack of real-world follow-through, and ignored operational constraints—and proposes a dual-pathway framework to improve real-world deployment through enhanced generalizability, targeted proofs of concept, and hybrid decision-making where DRL alone is insufficient.

ABSTRACT

Deep Reinforcement Learning (DRL) is considered a potential framework to improve many real-world autonomous systems; it has attracted the attention of multiple and diverse fields. Nevertheless, the successful deployment in the real world is a test most of DRL models still need to pass. In this work we focus on this issue by reviewing and evaluating the research efforts from both domain-agnostic and domain-specific communities. On one hand, we offer a comprehensive summary of DRL challenges and summarize the different proposals to mitigate them; this helps identifying five gaps of domain-agnostic research. On the other hand, from the domain-specific perspective, we discuss different success stories and argue why other models might fail to be deployed. Finally, we take up on ways to move forward accounting for both perspectives.

Motivation & Objective

• To analyze the disconnect between domain-agnostic DRL research and real-world deployment challenges.
• To identify key shortcomings in current DRL research that hinder real-world applicability, especially in operational contexts.
• To evaluate domain-specific DRL success stories and understand why many models fail to deploy despite strong performance in simulation.
• To propose a framework that aligns generalizable DRL advances with domain-specific operational needs.
• To guide future research toward more operationally viable DRL systems by addressing design tradeoffs and system-level constraints.

Proposed method

• Conducts a comprehensive review of domain-agnostic DRL challenges and mitigation strategies from recent literature.
• Identifies five critical gaps in domain-agnostic DRL research: robotics bias, insufficient work on combined challenges, lack of real-world validation, poor understanding of design tradeoffs, and omission of operational considerations.
• Analyzes domain-specific DRL applications across diverse fields (e.g., robotics, communications, healthcare, drug discovery) to highlight successful deployments and failure points.
• Proposes a three-pronged strategy for real-world deployment: improving generalizability of existing models, developing additional proofs of concept to cover operational space, and recognizing scenarios where DRL alone is infeasible.
• Integrates insights from both perspectives to advocate for hybrid decision-making systems where DRL complements other control or planning methods.
• Uses a conceptual framework (Figure 2) to distinguish between domain-specific operational contexts and generalizable DRL proofs of concept, emphasizing alignment between them.

Experimental results

Research questions

• RQ1 Why do many Deep Reinforcement Learning models fail to deploy in real-world environments despite strong performance in simulation?
• RQ2 What are the key gaps in domain-agnostic DRL research that prevent real-world applicability?
• RQ3 How do domain-specific DRL applications succeed or fail in real-world deployment, and what role do operational constraints play?
• RQ4 To what extent can generalizable DRL models cover the full operational space of real-world systems?
• RQ5 In which operational scenarios might DRL be insufficient, and how can it be combined with other decision-making frameworks?

Key findings

• Five major gaps were identified in domain-agnostic DRL research: over-reliance on robotics use cases, insufficient attention to combined challenges, lack of real-world validation, limited understanding of design tradeoffs, and neglect of operational considerations.
• Many domain-specific DRL successes are achieved through highly tailored models that sacrifice generalizability, making them unsuitable for broader operational deployment.
• Real-world deployment is often hindered not by algorithmic limitations alone, but by the absence of operational context in model design and evaluation.
• Improving generalizability is essential but insufficient; a portfolio of specialized DRL models may be necessary to cover complex operational spaces.
• In some cases, DRL alone cannot address the full operational spectrum, necessitating integration with non-DRL decision-making systems.
• The alignment of domain-agnostic research with domain-specific operational needs is critical for advancing real-world DRL deployment.