[论文解读] Offloading and Resource Allocation with General Task Graph in Mobile Edge Computing: A Deep Reinforcement Learning Approach

Yan Jia, Suzhi Bi+1 more|arXiv (Cornell University)|Feb 19, 2020

IoT and Edge/Fog Computing参考文献 29被引用 7

一句话总结

本文提出了一种深度强化学习（DRL）框架，用于在移动边缘计算中处理具有通用任务图的联合卸载与资源分配问题，利用具有低复杂度评论家的演员-评论家结构来评估能耗-时延成本（ETC）。该方法在显著降低计算复杂度的同时，实现了高达99.1%的最优性能。

ABSTRACT

In this paper, we consider a mobile-edge computing system, where an access point assists a mobile device (MD) to execute an application consisting of multiple tasks following a general task call graph. The objective is to jointly determine the offloading decision of each task and the resource allocation under time-varying wireless fading channels and stochastic edge computing capability, so that the energy-time cost (ETC) of the MD is minimized. Solving the problem is particularly hard due to the combinatorial offloading decisions and the strong coupling among task executions under the general dependency model. Conventional numerical optimization methods are inefficient to solve such a problem, especially when the problem size is large. To address the issue, we propose a deep reinforcement learning (DRL) framework based on the actor-critic learning structure. In particular, the actor network utilizes a DNN to learn the optimal mapping from the input states to the binary offloading decision of each task. Meanwhile, by analyzing the structure of the optimal solution, we derive a low-complexity algorithm for the critic network to quickly evaluate the ETC performance of the offloading decisions output by the actor network. With the low-complexity critic network, we can quickly select the best offloading action and subsequently store the state-action pair in an experience replay memory as the training dataset to continuously improve the action generation DNN. To further reduce the complexity, we show that the optimal offloading decision exhibits an one-climb structure, which can be utilized to significantly reduce the search space of action generation. Numerical results show that for various types of task graphs, the proposed algorithm achieves up to $99.1\%$ of the optimal performance while significantly reducing the computational complexity compared to the existing optimization methods.

研究动机与目标

• 解决在时间变化的无线信道和随机边缘计算能力下，移动边缘计算（MEC）系统中联合卸载与资源分配的挑战。
• 最小化移动设备在执行具有复杂依赖关系的通用任务图应用时的能耗-时延成本（ETC）。
• 克服传统优化方法效率低下且在大规模或动态场景中不切实际的局限性。
• 利用最优解的结构特性（如单次迁移策略），缩小动作空间，提升学习效率。

提出的方法

• 所提出的DRL框架采用演员-评论家架构，其中演员网络使用深度神经网络（DNN）将系统状态（信道增益和CPU频率）映射为二进制卸载决策。
• 评论家网络采用基于最优解结构分析推导出的低复杂度算法，快速评估卸载决策的ETC性能。
• 该框架使用经验回放机制，结合演员生成的状态-动作对与评论家的评估结果，迭代优化策略。
• 该方法利用单次迁移特性——即执行最多在设备与边缘之间迁移一次——显著缩小卸载动作的搜索空间。
• 评论家的评估函数设计为计算高效，即使在通用任务图中存在复杂依赖关系，也能实现实时训练与推理。

实验结果

研究问题

• RQ1 深度强化学习方法能否在动态无线条件下，有效处理通用任务图中组合性卸载决策与强任务依赖耦合问题？
• RQ2 如何设计评论家网络，以在不依赖计算成本高昂的优化求解器的情况下，实现快速且准确的ETC评估？
• RQ3 最优卸载策略中的单次迁移结构是否适用于通用任务图？该结构能否被用于缩小动作空间并提升学习效率？
• RQ4 与传统优化方法相比，所提出的DRL框架在保持低计算复杂度的前提下，能在多大程度上实现接近最优的ETC性能？

主要发现

• 所提出的DRL框架在各种任务图类型下，实现了高达99.1%的最优能耗-时延成本（ETC）性能，证明了其接近最优性。
• 低复杂度评论家网络实现了对卸载决策的快速评估，使训练过程相比传统数值优化方法显著更高效。
• 对单次迁移策略特性的利用显著缩小了卸载动作的搜索空间，加速了收敛并提升了样本效率。
• 该框架在性能和对动态信道与计算条件的适应能力方面，均优于现有的启发式方法与凸松弛方法。