[Paper Review] Graph Attention Network-based Multi-agent Reinforcement Learning for Slicing Resource Management in Dense Cellular Network

Yan Shao, Rongpeng Li+4 more|arXiv (Cornell University)|Aug 11, 2021

Software-Defined Networks and 5G4 citations

TL;DR

This paper proposes a Graph Attention Network (GAT)-enhanced multi-agent reinforcement learning (MARL) framework for dynamic network slicing resource management in dense 5G cellular networks. By modeling base stations as agents and leveraging GAT to capture temporal-spatial dependencies, the approach improves cooperation among base stations, achieving up to 7% higher system utility and superior convergence stability compared to baseline DRL methods like DQN and A2C.

ABSTRACT

Network slicing (NS) management devotes to providing various services to meet distinct requirements over the same physical communication infrastructure and allocating resources on demands. Considering a dense cellular network scenario that contains several NS over multiple base stations (BSs), it remains challenging to design a proper real-time inter-slice resource management strategy, so as to cope with frequent BS handover and satisfy the fluctuations of distinct service requirements. In this paper, we propose to formulate this challenge as a multi-agent reinforcement learning (MARL) problem in which each BS represents an agent. Then, we leverage graph attention network (GAT) to strengthen the temporal and spatial cooperation between agents. Furthermore, we incorporate GAT into deep reinforcement learning (DRL) and correspondingly design an intelligent real-time inter-slice resource management strategy. More specially, we testify the universal effectiveness of GAT for advancing DRL in the multi-agent system, by applying GAT on the top of both the value-based method deep Q-network (DQN) and a combination of policy-based and value-based method advantage actor-critic (A2C). Finally, we verify the superiority of the GAT-based MARL algorithms through extensive simulations.

Motivation & Objective

• Address the challenge of real-time inter-slice resource management in dense 5G cellular networks with fluctuating service demands and frequent base station handovers.
• Improve cooperation among base stations (BSs) to enhance system-wide resource allocation efficiency and meet diverse service requirements (eMBB, URLLC, mMTC).
• Develop a scalable, adaptive resource management strategy that dynamically allocates spectrum and computing resources across network slices.
• Verify the effectiveness of Graph Attention Networks (GAT) in enhancing deep reinforcement learning (DRL) performance in multi-agent settings for network slicing.

Proposed method

• Formulate the network slicing resource management problem as a multi-agent reinforcement learning (MARL) problem, where each base station (BS) acts as an independent agent.
• Integrate Graph Attention Networks (GAT) into the state representation to model spatial and temporal dependencies among BSs, enabling attention-based aggregation of neighboring agent states.
• Apply GAT as a preprocessing module before two mainstream DRL algorithms: Deep Q-Network (DQN) and Advantage Actor-Critic (A2C), creating GAT-DQN and GAT-A2C frameworks.
• Design a composite reward function that balances spectral efficiency (SE) and slice-specific reliability (SSR), with penalties for violating SLAs, especially for URLLC slices.
• Train the GAT-based MARL agents using centralized training with decentralized execution (CTDE), allowing full observability during training while preserving scalability during inference.
• Use graph-structured state inputs where nodes represent BSs and edges represent wireless connectivity or interference relationships, with GAT attention coefficients dynamically learned during training.

Experimental results

Research questions

• RQ1 Can GAT effectively model temporal and spatial dependencies among base stations to improve cooperation in multi-agent reinforcement learning for network slicing?
• RQ2 How does incorporating GAT into DRL frameworks (DQN and A2C) affect the convergence speed, stability, and final performance of inter-slice resource allocation?
• RQ3 To what extent does the GAT-based MARL approach outperform traditional hard slicing and vanilla DRL methods in terms of system utility, spectral efficiency (SE), and slice service reliability (SSR)?
• RQ4 How does the performance of GAT-enhanced MARL scale with increasing action space size and varying service demand granularity?
• RQ5 Does the GAT mechanism improve robustness and adaptability in dynamic, high-mobility network environments with fluctuating service demands?

Key findings

• GAT-DQN and GAT-A2C achieve up to 7% higher system utility than baseline DQN and A2C, respectively, particularly in large action space scenarios (Δ = 0.18 MHz), demonstrating the scalability of the GAT-enhanced approach.
• The GAT-based algorithms improve spectral efficiency (SE) by up to 7% while maintaining or exceeding SLA requirements for URLLC slices (SSR ≥ 0.9), indicating effective trade-off between performance and reliability.
• GAT-A2C shows more stable convergence behavior compared to GAT-DQN, especially in fine-grained resource allocation (Δ = 0.18 MHz), suggesting better policy stability in complex environments.
• The integration of GAT leads to a 4% improvement in utility over DQN in coarse-grained scenarios (Δ = 0.54 MHz), confirming its consistent benefit across different action space sizes.
• All DRL-based algorithms, especially GAT-enhanced ones, significantly improve SSR for URLLC services (0.8–0.9) without degrading performance for eMBB or mMTC (SSR ≈ 1.0), ensuring strict SLA compliance.
• The method remains effective across different hyper-parameter settings (e.g., β = [1,1,1], [1,2,3], [1,1,5]), showing robustness to parameter variations and minimal need for manual tuning.