[Paper Review] Deep Reinforcement Learning Based Dynamic Trajectory Control for UAV-assisted Mobile Edge Computing

Liang Wang, Kezhi Wang+4 more|arXiv (Cornell University)|Nov 10, 2019

UAV Applications and Optimization44 references16 citations

TL;DR

This paper proposes a deep reinforcement learning-based trajectory control algorithm (RAT) for UAV-assisted mobile edge computing, jointly optimizing user association, resource allocation, and UAV trajectories to minimize total UE energy consumption. Unlike convex optimization-based CAT, RAT enables real-time, adaptive decision-making across diverse UAV takeoff points with faster convergence and superior performance, validated through simulations in 2D and 3D scenarios.

ABSTRACT

In this paper, we consider a platform of flying mobile edge computing (F-MEC), where unmanned aerial vehicles (UAVs) serve as equipment providing computation resource, and they enable task offloading from user equipment (UE). We aim to minimize energy consumption of all the UEs via optimizing the user association, resource allocation and the trajectory of UAVs. To this end, we first propose a Convex optimizAtion based Trajectory control algorithm (CAT), which solves the problem in an iterative way by using block coordinate descent (BCD) method. Then, to make the real-time decision while taking into account the dynamics of the environment (i.e., UAV may take off from different locations), we propose a deep Reinforcement leArning based Trajectory control algorithm (RAT). In RAT, we apply the Prioritized Experience Replay (PER) to improve the convergence of the training procedure. Different from the convex optimization based algorithm which may be susceptible to the initial points and requires iterations, RAT can be adapted to any taking off points of the UAVs and can obtain the solution more rapidly than CAT once training process has been completed. Simulation results show that the proposed CAT and RAT achieve the similar performance and both outperform traditional algorithms.

Motivation & Objective

• Address the challenge of minimizing long-term energy consumption in UAV-assisted mobile edge computing (F-MEC) with dynamic, continuous decision variables.
• Overcome limitations of traditional convex optimization methods—such as sensitivity to initial points and high iteration overhead—by enabling real-time, adaptive control.
• Design a scalable solution for multi-UAV systems where user association, resource allocation, and trajectory control are jointly optimized.
• Enable rapid, generalizable decision-making from any UAV takeoff location, improving flexibility and deployment adaptability.
• Demonstrate performance superiority of the proposed method over conventional and heuristic baselines in energy efficiency and convergence speed.

Proposed method

• Propose a Convex Optimization-based Trajectory control (CAT) algorithm using block coordinate descent (BCD) to iteratively solve for UAV trajectories, user association, and resource allocation.
• Develop a Deep Reinforcement Learning-based Trajectory control (RAT) framework using two deep Q-networks (DQN): an actor network for action selection (direction and distance) and a critic network for value evaluation.
• Integrate a low-complexity matching algorithm to dynamically assign users and allocate resources based on UAV availability and channel conditions.
• Employ Prioritized Experience Replay (PER) to accelerate training convergence by focusing on high-impact experiences.
• Define the overall energy consumption of all UEs as the reinforcement learning reward signal to guide policy learning.
• Train RAT in a 3D environment with realistic UAV dynamics, including altitude changes and spatial mobility, to reflect real-world deployment constraints.

Experimental results

Research questions

• RQ1 Can a deep reinforcement learning framework effectively optimize UAV trajectories, user association, and resource allocation in a dynamic, multi-UAV F-MEC environment?
• RQ2 How does the proposed RAT algorithm perform in terms of energy consumption and convergence speed compared to convex optimization-based CAT and heuristic baselines?
• RQ3 To what extent can RAT generalize across different UAV takeoff locations without retraining?
• RQ4 How does the inclusion of altitude control in the 3D trajectory model affect system performance and energy efficiency?
• RQ5 What is the impact of increasing the number of UAVs on the overall energy consumption of UEs in the proposed framework?

Key findings

• RAT achieves the lowest overall energy consumption among all evaluated algorithms, consistently outperforming CAT, CM, RM, and LE across all tested scenarios.
• In a 3D scenario with two UAVs, RAT reduces UE energy consumption to approximately 350 J after training, compared to 600–700 J at the start of learning.
• With increasing UAV count, the overall energy consumption of UEs decreases for all algorithms due to higher computational capacity, but RAT maintains the best performance margin.
• RAT demonstrates superior adaptability, achieving fast inference and stable performance regardless of UAV initial takeoff locations, unlike CAT which is sensitive to initialization.
• The 3D trajectory results show UAVs initially ascend to increase coverage, then dynamically reposition to serve multiple UEs, balancing data rate and service range.
• In comparison to CM (constant altitude) and RM (random movement), RAT reduces UAV energy consumption by over 10% and UE energy consumption by more than 20% on average.