[Paper Review] Reinforcement Learning Framework for Server Placement and Workload Allocation in Multi-Access Edge Computing

Anahita Mazloomi, Hani Sami+3 more|arXiv (Cornell University)|Feb 21, 2022

IoT and Edge/Fog Computing4 citations

TL;DR

This paper proposes a novel reinforcement learning framework, TDMC/QMC, for joint edge server placement and base station allocation in multi-access edge computing (MEC) to minimize network delay and the number of servers. By designing an efficient Markov Decision Process with optimized state/action spaces and penalty functions, the method enables stable Temporal Difference learning (TD(λ)) and Q-learning, outperforming benchmarks in cost reduction on a real-world Shanghai dataset.

ABSTRACT

Cloud computing is a reliable solution to provide distributed computation power. However, real-time response is still challenging regarding the enormous amount of data generated by the IoT devices in 5G and 6G networks. Thus, multi-access edge computing (MEC), which consists of distributing the edge servers in the proximity of end-users to have low latency besides the higher processing power, is increasingly becoming a vital factor for the success of modern applications. This paper addresses the problem of minimizing both, the network delay, which is the main objective of MEC, and the number of edge servers to provide a MEC design with minimum cost. This MEC design consists of edge servers placement and base stations allocation, which makes it a joint combinatorial optimization problem (COP). Recently, reinforcement learning (RL) has shown promising results for COPs. However, modeling real-world problems using RL when the state and action spaces are large still needs investigation. We propose a novel RL framework with an efficient representation and modeling of the state space, action space and the penalty function in the design of the underlying Markov Decision Process (MDP) for solving our problem.

Motivation & Objective

• To address the joint optimization of edge server placement and base station allocation in MEC to minimize network delay and reduce the number of servers.
• To overcome challenges in applying reinforcement learning (RL) to large-scale, real-world combinatorial optimization problems (COPs) with complex state and action spaces.
• To design an efficient Markov Decision Process (MDP) with precise state representation, action space modeling, and penalty function to enable stable TD(λ) and Q-learning learning.
• To evaluate the performance of the proposed TDMC and QMC algorithms against benchmarks like K-means, GA, and DQN in minimizing total network design cost.
• To investigate the impact of hyperparameters α and γ on learning convergence and solution quality.

Proposed method

• Proposes a custom MDP formulation with a compact state space representation that captures base station locations, server availability, and workload distribution.
• Defines a dynamic action space where agents select server placements and base station assignments in a discrete, combinatorial manner.
• Introduces a multi-objective penalty function that balances network delay and server count to guide learning toward cost-minimizing solutions.
• Employs TD(λ) with eligibility traces (TDMC) and Q-learning (QMC, when λ=0) to stabilize learning and reduce overestimation bias in value function approximation.
• Uses a real-world dataset from Shanghai Telecommunication to simulate realistic base station and request distributions for training and evaluation.
• Evaluates deep RL via DQN to contrast performance with tabular Q-learning, revealing that non-linear approximation is unnecessary and harmful in this MDP design.

Experimental results

Research questions

• RQ1 Can a reinforcement learning framework effectively solve the joint edge server placement and base station allocation problem in MEC with large state and action spaces?
• RQ2 How does the design of the state space, action space, and penalty function impact the convergence and performance of TD(λ) and Q-learning in this COP?
• RQ3 What is the optimal configuration of learning rate (α) and discount factor (γ) for minimizing total network cost in this MEC setting?
• RQ4 Does deep Q-learning (DQN) outperform tabular Q-learning in this problem, or does the non-linear function approximation introduce instability and suboptimal policies?
• RQ5 How do the proposed TDMC and QMC algorithms compare to traditional benchmarks like K-means, GA, and random assignment in terms of cost, delay, and server count?

Key findings

• TDMC and QMC outperform all benchmarks, including GA and K-means, in minimizing total network cost, with QMC showing a slight edge in performance.
• The minimum cost achieved with α=0.4 and γ=0.9 was 2967.9766, which was the lowest across all hyperparameter combinations tested.
• DQN failed to converge or stabilize, producing higher and more variable costs than TDMC and QMC, due to unnecessary non-linear function approximation in a problem that benefits from tabular methods.
• The efficient MDP design, including compact state representation and a well-structured penalty function, enabled stable learning and convergence despite the large-scale nature of the problem.
• The joint optimization of delay and server count using TDMC/QMC achieved a better trade-off than methods that optimize only one objective, such as K-means or Top-K.
• The study confirms that hyperparameters α and γ significantly affect learning performance, with α=0.4 and γ=0.9 yielding the best balance between convergence speed and solution quality.