[Paper Review] A Hierarchical Framework of Cloud Resource Allocation and Power Management Using Deep Reinforcement Learning

Ning Liu, Zhe Li+6 more|arXiv (Cornell University)|Mar 13, 2017

IoT and Edge/Fog Computing54 citations

TL;DR

The paper proposes a hierarchical DRL framework with a global DRL-based VM allocation tier and a local RL-based power management tier, using autoencoder/weight sharing to handle high-dimensional state spaces and LSTM workload prediction to improve DPM.

ABSTRACT

Automatic decision-making approaches, such as reinforcement learning (RL), have been applied to (partially) solve the resource allocation problem adaptively in the cloud computing system. However, a complete cloud resource allocation framework exhibits high dimensions in state and action spaces, which prohibit the usefulness of traditional RL techniques. In addition, high power consumption has become one of the critical concerns in design and control of cloud computing systems, which degrades system reliability and increases cooling cost. An effective dynamic power management (DPM) policy should minimize power consumption while maintaining performance degradation within an acceptable level. Thus, a joint virtual machine (VM) resource allocation and power management framework is critical to the overall cloud computing system. Moreover, novel solution framework is necessary to address the even higher dimensions in state and action spaces. In this paper, we propose a novel hierarchical framework for solving the overall resource allocation and power management problem in cloud computing systems. The proposed hierarchical framework comprises a global tier for VM resource allocation to the servers and a local tier for distributed power management of local servers. The emerging deep reinforcement learning (DRL) technique, which can deal with complicated control problems with large state space, is adopted to solve the global tier problem. Furthermore, an autoencoder and a novel weight sharing structure are adopted to handle the high-dimensional state space and accelerate the convergence speed. On the other hand, the local tier of distributed server power managements comprises an LSTM based workload predictor and a model-free RL based power manager, operating in a distributed manner.

Motivation & Objective

• Motivate joint VM resource allocation and dynamic power management (DPM) in cloud data centers.
• Develop a scalable hierarchical architecture with a global DRL-based allocator and a distributed local power manager.
• Reduce state/action space size and improve convergence using autoencoder and shared weights.
• Predict future workloads with LSTM to enhance DPM decisions.
• Demonstrate power/energy savings and latency trade-offs on real traces.

Proposed method

• Adopt a two-tier hierarchy: a global DRL-based VM allocator and a local distributed power manager for servers.
• Use continuous-time, event-driven decisions at VM arrival epochs to keep actions enumerable.
• Employ an offline DNN (with autoencoder and weight sharing) to estimate Q-values for action selection.
• Apply online deep Q-learning (SMDP) with epsilon-greedy policy for the global tier.
• In the local tier, use an LSTM workload predictor plus a model-free RL power manager for ON/OFF decisions.
• Define a reward incorporating total power, VM count, and reliability as a linear combination to guide learning.

Experimental results

Research questions

• RQ1 Can a hierarchical DRL framework scale to high-dimensional cloud resource allocation problems while maintaining performance guarantees?
• RQ2 Does integrating an autoencoder with weight sharing improve DRL convergence and scalability for VM allocation?
• RQ3 How effective is LSTM-based workload prediction in enhancing distributed DPM decisions?
• RQ4 What are the gains in power/energy savings and latency trade-offs when using the proposed framework on real workload traces?

Key findings

• The hierarchical framework achieves significant power and energy savings over baselines on Google cluster traces (53.97% in a 30-server cluster with 95,000 job requests).
• The framework provides the best trade-off between latency and power/energy consumption in server clusters.
• Average per-job latency savings reach up to 16.16% with the same energy usage, and average power/energy savings reach up to 16.20% with the same latency.
• DRL-based global tier has low online computational complexity, scaling with the number of servers at each decision epoch.
• An autoencoder with a weight-sharing scheme accelerates convergence and enables scalable Q-value estimation across server groups.
• Local tier combines LSTM-based workload prediction with a model-free RL power manager to implement distributed DPM decisions.