[Paper Review] Deep Reinforcement Learning for Scheduling in Cellular Networks

Jian Wang, Xu Chen+4 more|arXiv (Cornell University)|May 15, 2019

Advanced MIMO Systems Optimization10 references4 citations

TL;DR

This paper proposes a deep reinforcement learning (DRL)-based scheduler for cellular networks that integrates expert knowledge to improve training efficiency and performance. Instead of replacing the conventional proportional fair (PF) scheduler, the authors introduce an AI scheduler that learns from the PF algorithm or from a dual DRL agent, achieving faster convergence and near-optimal fairness-throughput trade-offs compared to direct DRL training.

ABSTRACT

Integrating artificial intelligence (AI) into wireless networks has drawn significant interest in both industry and academia. A common solution is to replace partial or even all modules in the conventional systems, which is often lack of efficiency and robustness due to their ignoring of expert knowledge. In this paper, we take deep reinforcement learning (DRL) based scheduling as an example to investigate how expert knowledge can help with AI module in cellular networks. A simulation platform, which has considered link adaption, feedback and other practical mechanisms, is developed to facilitate the investigation. Besides the traditional way, which is learning directly from the environment, for training DRL agent, we propose two novel methods, i.e., learning from a dual AI module and learning from the expert solution. The results show that, for the considering scheduling problem, DRL training procedure can be improved on both performance and convergence speed by involving the expert knowledge. Hence, instead of replacing conventional scheduling module in the system, adding a newly introduced AI module, which is capable to interact with the conventional module and provide more flexibility, is a more feasible solution.

Motivation & Objective

• To investigate how expert knowledge can enhance DRL training in wireless network scheduling.
• To compare direct DRL training with dual-agent and expert-knowledge-based training methods.
• To evaluate whether adding an AI scheduler is more feasible than replacing the conventional scheduler.
• To develop a realistic simulation platform incorporating practical mechanisms like AMC and feedback.
• To demonstrate that AI-scheduler integration can improve system flexibility and adaptability without compromising stability.

Proposed method

• A simulation platform is built to model real-world cellular network dynamics, including link adaptation, feedback, and channel variations.
• Three training methods are evaluated: direct DRL training, dual DRL agent training (mutual learning), and expert-knowledge distillation using the PF algorithm.
• The DRL agent uses a dueling DQN architecture with 320-unit fully connected layers and ReLU activation; softmax is used in the actor network.
• The reward function balances throughput and fairness using weighted coefficients α and β, with performance normalized against the PF baseline.
• Training is conducted across 28 parallel simulators to improve convergence and generalization.
• The PF algorithm is used as a reference expert policy to guide the DRL agent via imitation learning in the expert-learning method.

Experimental results

Research questions

• RQ1 Can DRL effectively learn scheduling policies in a realistic cellular network environment with practical constraints?
• RQ2 How does incorporating expert knowledge (e.g., PF algorithm) affect DRL training convergence and performance?
• RQ3 Does dual-agent training improve convergence compared to single-agent direct learning?
• RQ4 Can an AI scheduler be trained online while the conventional PF scheduler remains active, ensuring system stability?
• RQ5 What is the trade-off between throughput and fairness in DRL-based scheduling, and how can it be optimized?

Key findings

• Direct DRL training converges in ~1000 updates but struggles to match the fairness of the PF algorithm, especially under non-stationary channel conditions.
• Dual-learning with two mutually training agents achieves convergence to near-optimal performance after ~2000 updates, avoiding local optima but with slower training.
• Expert-knowledge distillation enables the DRL agent to converge to near-optimal performance in ~1000 updates and fully match PF performance by ~4000 updates.
• The expert-learning method achieves the best balance between training speed and final performance, significantly outperforming direct learning in convergence and fairness.
• The results support the feasibility of augmenting conventional schedulers with an AI module rather than replacing them, ensuring robustness and adaptability.
• The study demonstrates that integrating expert knowledge into DRL training enhances both efficiency and performance in wireless scheduling.