[Paper Review] A View on Deep Reinforcement Learning in System Optimization

Ameer Haj-Ali, Nesreen K. Ahmed+4 more|arXiv (Cornell University)|Aug 4, 2019

Reinforcement Learning in Robotics77 references18 citations

TL;DR

This paper reviews deep reinforcement learning (DRL) applications in system optimization, proposing standardized metrics to evaluate DRL efficacy. It analyzes challenges in state/action/reward formulation, model efficiency, and robustness, advocating for better benchmarks and encoding to improve generalization and performance in real-world systems like cloud scheduling and compiler optimization.

ABSTRACT

Many real-world systems problems require reasoning about the long term consequences of actions taken to configure and manage the system. These problems with delayed and often sequentially aggregated reward, are often inherently reinforcement learning problems and present the opportunity to leverage the recent substantial advances in deep reinforcement learning. However, in some cases, it is not clear why deep reinforcement learning is a good fit for the problem. Sometimes, it does not perform better than the state-of-the-art solutions. And in other cases, random search or greedy algorithms could outperform deep reinforcement learning. In this paper, we review, discuss, and evaluate the recent trends of using deep reinforcement learning in system optimization. We propose a set of essential metrics to guide future works in evaluating the efficacy of using deep reinforcement learning in system optimization. Our evaluation includes challenges, the types of problems, their formulation in the deep reinforcement learning setting, embedding, the model used, efficiency, and robustness. We conclude with a discussion on open challenges and potential directions for pushing further the integration of reinforcement learning in system optimization.

Motivation & Objective

• To address the lack of standardized evaluation metrics for deep reinforcement learning (DRL) in system optimization tasks.
• To identify and analyze core challenges in applying DRL to real-world systems, including state/action/reward formulation and environment dynamics.
• To evaluate the effectiveness of DRL across diverse system optimization problems such as scheduling, congestion control, and compiler optimization.
• To propose a framework of essential metrics for assessing DRL performance in terms of efficiency, robustness, and generalization.
• To guide future research toward more stable, transferable, and scalable DRL solutions in complex system environments.

Proposed method

• Proposes a set of quintessential evaluation metrics for DRL in system optimization, focusing on efficiency, robustness, and generalization.
• Reviews prior work in DRL for system optimization, including packet classification, congestion control, and job scheduling in cloud environments.
• Analyzes state, action, and reward design in DRL formulations, emphasizing the importance of meaningful and representative encodings.
• Discusses model-based and off-policy DRL algorithms, including DQN, PPO, and SAC, in the context of system control and optimization.
• Introduces mechanisms to prevent infinite loops in DRL agents, such as limiting action repetitions and selecting suboptimal actions when no state change occurs.
• Advocates for transfer learning and better state representation to improve sample efficiency and generalization across system workloads.

Experimental results

Research questions

• RQ1 What are the key challenges in formulating system optimization problems as deep reinforcement learning tasks?
• RQ2 Why do some DRL solutions underperform compared to simpler baselines like random search or greedy algorithms?
• RQ3 How can we evaluate the effectiveness of DRL in system optimization beyond anecdotal or task-specific results?
• RQ4 What role do state, action, and reward design play in the success or failure of DRL in real-world systems?
• RQ5 How can DRL be made more robust and generalizable across varying system workloads and dynamic environments?

Key findings

• Standardized evaluation metrics for DRL in system optimization are currently lacking, leading to inconsistent and hard-to-compare results across studies.
• DRL performance is highly sensitive to the design of state, action, and reward representations; poor encoding leads to suboptimal or unstable learning.
• In some cases, simpler baselines such as greedy algorithms or random search outperform DRL, indicating that DRL is not universally superior.
• Infinite loops can occur in DRL agents when actions fail to change the environment state, which can be mitigated by limiting action repetitions and selecting alternative actions.
• Transfer learning and better state representation are critical for improving sample efficiency and generalization in dynamic system environments.
• DRL shows strong potential in complex system tasks such as cloud job scheduling, compiler optimization, and network congestion control, but requires more robust benchmarks and frameworks.