[Paper Review] Deep Reinforcement Learning for URLLC data management on top of scheduled eMBB traffic

Fabio Saggese, Luca Pasqualini+2 more|arXiv (Cornell University)|Jan 1, 2021

Wireless Communication Security TechniquesEngineering17 references25 citations

TL;DR

The paper proposes a PPO-based DRL agent to dynamically puncture eMBB resources to accommodate URLLC traffic while meeting URLLC latency and reliability constraints.

ABSTRACT

With the advent of 5G and the research into beyond 5G (B5G) networks, a novel and very relevant research issue is how to manage the coexistence of different types of traffic, each with very stringent but completely different requirements. In this paper we propose a deep reinforcement learning (DRL) algorithm to slice the available physical layer resources between ultra-reliable low-latency communications (URLLC) and enhanced Mobile BroadBand (eMBB) traffic. Specifically, in our setting the time-frequency resource grid is fully occupied by eMBB traffic and we train the DRL agent to employ proximal policy optimization (PPO), a state-of-the-art DRL algorithm, to dynamically allocate the incoming URLLC traffic by puncturing eMBB codewords. Assuming that each eMBB codeword can tolerate a certain limited amount of puncturing beyond which is in outage, we show that the policy devised by the DRL agent never violates the latency requirement of URLLC traffic and, at the same time, manages to keep the number of eMBB codewords in outage at minimum levels, when compared to other state-of-the-art schemes.

Motivation & Objective

• Address the coexistence of URLLC and eMBB traffic on a shared resource grid.
• Develop a DRL-based scheduler that decides when and where to transmit URLLC packets without violating URLLC latency.
• Enable some slack in URLLC scheduling to reduce impact on eMBB by learning puncturing policies.
• Model eMBB codewords with puncturing tolerance to sustain reliability under URLLC puncturing.

Proposed method

• Model the system as a Markov decision process with states capturing URLLC queue, latency slack, and eMBB codeword outages.
• Use Proximal Policy Optimization (PPO) as a policy gradient DRL method to learn the transmission decision.
• Represent actions as choosing a frequency resource for URLLC transmission or opting not to transmit in a minislot.
• Define a reward that penalizes eMBB outages and URLLC latency violations to balance the two slices.
• Provide a neural network architecture with separate policy and value networks for PPO.

Experimental results

Research questions

• RQ1 Can a DRL-based scheduler learn to puncture eMBB resources to accommodate URLLC without violating URLLC latency requirements?
• RQ2 How does the PPO-trained agent compare to heuristic schemes in terms of URLLC latency guarantees and eMBB outage rates?
• RQ3 Does the learned policy generalize across different URLLC arrival rates and codeword puncturing configurations?

Key findings

• The PPO agent outperforms heuristic schemes across evaluated URLLC arrival rates in terms of overall reward.
• The PPO policy keeps URLLC latency constraints satisfied while minimizing eMBB codeword outages.
• The learned policy demonstrates generalization to different episode lengths and codeword distributions without retraining.
• Random schemes may occasionally show lower outage percentages due to latency violations, which PPO avoids.
• PPO maintains strong performance even when codewords have heterogeneous puncturing tolerances (C_w in {0,1}).
• Results indicate the approach scales to longer horizons without retraining.