[Paper Review] When Multiple Agents Learn to Schedule: A Distributed Radio Resource Management Framework

Navid Naderializadeh, Jaroslaw J. Sydir+3 more|arXiv (Cornell University)|Jun 20, 2019

Advanced MIMO Systems Optimization15 references4 citations

TL;DR

This paper proposes a multi-agent deep reinforcement learning (DRL) framework for distributed radio resource management in dense wireless networks, where each access point acts as a DQN agent making link scheduling decisions based on local and delayed neighboring observations. The framework achieves a strong trade-off between average and 5th percentile user throughput, outperforming decentralized baselines and approaching centralized exhaustive search performance while demonstrating robustness to changes in network density.

ABSTRACT

Interference among concurrent transmissions in a wireless network is a key factor limiting the system performance. One way to alleviate this problem is to manage the radio resources in order to maximize either the average or the worst-case performance. However, joint consideration of both metrics is often neglected as they are competing in nature. In this article, a mechanism for radio resource management using multi-agent deep reinforcement learning (RL) is proposed, which strikes the right trade-off between maximizing the average and the $5^{th}$ percentile user throughput. Each transmitter in the network is equipped with a deep RL agent, receiving partial observations from the network (e.g., channel quality, interference level, etc.) and deciding whether to be active or inactive at each scheduling interval for given radio resources, a process referred to as link scheduling. Based on the actions of all agents, the network emits a reward to the agents, indicating how good their joint decisions were. The proposed framework enables the agents to make decisions in a distributed manner, and the reward is designed in such a way that the agents strive to guarantee a minimum performance, leading to a fair resource allocation among all users across the network. Simulation results demonstrate the superiority of our approach compared to decentralized baselines in terms of average and $5^{th}$ percentile user throughput, while achieving performance close to that of a centralized exhaustive search approach. Moreover, the proposed framework is robust to mismatches between training and testing scenarios. In particular, it is shown that an agent trained on a network with low transmitter density maintains its performance and outperforms the baselines when deployed in a network with a higher transmitter density.

Motivation & Objective

• To address the challenge of interference and unfair resource allocation in ultra-dense wireless networks.
• To design a distributed radio resource management mechanism that fairly balances average and minimum user throughput.
• To develop a scalable and robust learning-based scheduling framework resilient to mismatches between training and deployment environments.
• To enable practical deployment of DRL agents in real-world networks through a hybrid centralized training and distributed inference architecture.

Proposed method

• Each access point (AP) is equipped with a deep Q-network (DQN) agent that makes scheduling decisions based on partial observations of local channel quality and interference levels.
• Agents receive delayed and infrequent observations from neighboring APs to reflect real-world communication constraints.
• A centralized training framework aggregates experiences from all agents and uses a reward function that prioritizes both high average throughput and a minimum 5th percentile user rate to ensure fairness.
• The training process uses a replay buffer and periodic weight updates from a central entity to stabilize learning and improve convergence.
• The framework employs a three-tiered communication architecture: real-time inference links, medium-frequency training data collection, and low-frequency policy update and pause/termination signals.
• The system is designed to allow a single trained policy to generalize across different network densities without retraining.

Experimental results

Research questions

• RQ1 Can a multi-agent DRL framework achieve a balanced trade-off between average and minimum user throughput in interference-limited wireless networks?
• RQ2 How well does a DRL agent trained in a low-density network generalize to higher-density deployment scenarios?
• RQ3 Can a centralized training with distributed inference approach maintain performance and fairness under dynamic and heterogeneous network conditions?
• RQ4 What are the practical challenges of deploying DRL-based scheduling in real-world wireless networks, and how can they be mitigated?

Key findings

• The proposed DRL framework outperforms both full reuse and time-division multiplexing (TDM) baselines in both average and 5th percentile user throughput.
• The framework achieves performance close to that of a centralized exhaustive search, demonstrating near-optimal efficiency in resource allocation.
• An agent trained on a network with four APs maintains strong performance when deployed in a network with up to ten APs, indicating high robustness to network density changes.
• The framework is robust to distributional shift, maintaining effective performance even when testing scenarios differ from training conditions in terms of transmitter density and channel conditions.
• The use of delayed and infrequent neighbor observations enables practical deployment while preserving learning effectiveness.